[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:29 dist-test-slave-rfd5 krb5kdc[31858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:29 dist-test-slave-rfd5 krb5kdc[31858](info): set up 2 sockets
Sep 27 18:44:29 dist-test-slave-rfd5 krb5kdc[31858](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 18:44:30.034466 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:30 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:30.204921 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44079
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:30.533595 31871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:30.533926 31871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:30.534097 31871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:30.542028 31871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:30.542188 31871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:30.542285 31871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:30.542395 31871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:30.542500 31871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:30.553577 31871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:30.555034 31871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:30.556480 31871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:30.569752 31877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:30.569736 31879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:30.569854 31876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:30.570569 31871 server_base.cc:1034] running on GCE node
I20260927 18:44:30.571432 31871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:30.572944 31871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:30.574147 31871 hybrid_clock.cc:648] HybridClock initialized: now 1790534670574099 us; error 44 us; skew 500 ppm
I20260927 18:44:30.574519 31871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:30 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534670, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:30.577855 31871 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:30.579404 31871 webserver.cc:460] Webserver started at http://127.31.26.254:42363/ using document root <none> and password file <none>
I20260927 18:44:30.580142 31871 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:30.580286 31871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:30.580598 31871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:30.583731 31871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "f07bc24e52574d869b03373bb0a37133"
format_stamp: "Formatted at 2026-09-27 18:44:30 on dist-test-slave-rfd5"
server_key: "3fa64052b24761c30907b2580169dc51"
server_key_iv: "bef2b915be4e4c8b870471639b6ec82b"
server_key_version: "encryptionkey@0"
I20260927 18:44:30.584506 31871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "f07bc24e52574d869b03373bb0a37133"
format_stamp: "Formatted at 2026-09-27 18:44:30 on dist-test-slave-rfd5"
server_key: "3fa64052b24761c30907b2580169dc51"
server_key_iv: "bef2b915be4e4c8b870471639b6ec82b"
server_key_version: "encryptionkey@0"
I20260927 18:44:30.589936 31871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:44:30.594084 31886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:30.595582 31871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 18:44:30.595790 31871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "f07bc24e52574d869b03373bb0a37133"
format_stamp: "Formatted at 2026-09-27 18:44:30 on dist-test-slave-rfd5"
server_key: "3fa64052b24761c30907b2580169dc51"
server_key_iv: "bef2b915be4e4c8b870471639b6ec82b"
server_key_version: "encryptionkey@0"
I20260927 18:44:30.596045 31871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:30.616161 31871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:30.620832 31871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:30.621168 31871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:30.641258 31871 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44079
I20260927 18:44:30.641253 31937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44079 every 8 connection(s)
I20260927 18:44:30.642700 31871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:30.646741 31938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:30.652189 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 31871
I20260927 18:44:30.652526 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:30.654548 31851 external_mini_cluster.cc:1428] Setting key 158c6a78986d4be9232d98722b43f67b
I20260927 18:44:30.658085 31938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133: Bootstrap starting.
I20260927 18:44:30.662060 31938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:30.663116 31938 log.cc:824] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:30.666483 31938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133: No bootstrap required, opened a new log
I20260927 18:44:30.675211 31938 raft_consensus.cc:348] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07bc24e52574d869b03373bb0a37133" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44079 } }
I20260927 18:44:30.675625 31938 raft_consensus.cc:374] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:30.675725 31938 raft_consensus.cc:729] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07bc24e52574d869b03373bb0a37133, State: Initialized, Role: FOLLOWER
I20260927 18:44:30.676441 31938 consensus_queue.cc:260] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07bc24e52574d869b03373bb0a37133" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44079 } }
I20260927 18:44:30.676697 31938 raft_consensus.cc:388] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:30.676831 31938 raft_consensus.cc:482] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:30.677016 31938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:30.679574 31938 raft_consensus.cc:504] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07bc24e52574d869b03373bb0a37133" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44079 } }
I20260927 18:44:30.680107 31938 leader_election.cc:302] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f07bc24e52574d869b03373bb0a37133; no voters:
I20260927 18:44:30.680756 31938 leader_election.cc:288] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:30.680887 31943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:30.682193 31943 raft_consensus.cc:686] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [term 1 LEADER]: Becoming Leader. State: Replica: f07bc24e52574d869b03373bb0a37133, State: Running, Role: LEADER
I20260927 18:44:30.682780 31943 consensus_queue.cc:237] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07bc24e52574d869b03373bb0a37133" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44079 } }
I20260927 18:44:30.683199 31938 sys_catalog.cc:564] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:30.686252 31945 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f07bc24e52574d869b03373bb0a37133. Latest consensus state: current_term: 1 leader_uuid: "f07bc24e52574d869b03373bb0a37133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07bc24e52574d869b03373bb0a37133" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44079 } } }
I20260927 18:44:30.686672 31945 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [sys.catalog]: This master's current role is: LEADER
Sep 27 18:44:30 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:30.687394 31944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f07bc24e52574d869b03373bb0a37133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07bc24e52574d869b03373bb0a37133" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44079 } } }
I20260927 18:44:30.687732 31944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:30.701107 31948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:30.703810 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:30.658730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.31.26.254:44079)
0927 18:44:30.659664 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:30.659684 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:30.660829 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 18:44:30.671305 (+ 10476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:30.671330 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:30.671799 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:30.672250 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:30.675839 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:30.677824 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:30.679549 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:30.680400 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:30.680596 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:30.689454 (+ 8858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:30.689472 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:30.689526 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:30.689596 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:30.694914 (+ 5318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:30.696525 (+ 1611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:30.696540 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:30.696553 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:30.696863 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:30.697634 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:30.697642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:30.697648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:30.698592 (+ 944us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:30.698948 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:30.701101 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0927 18:44:30.701869 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":439,"threads_started":1}
I20260927 18:44:30.713258 31948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:30.722919 31948 catalog_manager.cc:1347] Generated new cluster ID: 22abbd0b598c4b5ca04be0a21a00f48c
I20260927 18:44:30.723089 31948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:30.752028 31948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:30.753571 31948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:30.759624 31948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f07bc24e52574d869b03373bb0a37133: Generated new TSK 0
I20260927 18:44:30.761153 31948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:44:30.808949 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44079
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:31.149721 31965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:31.150027 31965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:31.150210 31965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:31.157537 31965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:31.157689 31965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:31.157861 31965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:44:31.168241 31965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:31.169625 31965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:31.170956 31965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:31.184170 31973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:31.184870 31970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:31.184180 31971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:31.185508 31965 server_base.cc:1034] running on GCE node
I20260927 18:44:31.186137 31965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:31.187047 31965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:31.188277 31965 hybrid_clock.cc:648] HybridClock initialized: now 1790534671188215 us; error 61 us; skew 500 ppm
I20260927 18:44:31.188594 31965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:31 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:31.191758 31965 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:44:31.193357 31965 webserver.cc:460] Webserver started at http://127.31.26.193:43181/ using document root <none> and password file <none>
I20260927 18:44:31.194051 31965 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:31.194180 31965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:31.194506 31965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:31.198100 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "c70d7e500113446787fbf4ea3b073ebd"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-rfd5"
server_key: "8022d1fa7bf0ebdc121a883bdf9ee0e4"
server_key_iv: "e074c3b802c0f9ff21a58aafd8a01c46"
server_key_version: "encryptionkey@0"
I20260927 18:44:31.199083 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "c70d7e500113446787fbf4ea3b073ebd"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-rfd5"
server_key: "8022d1fa7bf0ebdc121a883bdf9ee0e4"
server_key_iv: "e074c3b802c0f9ff21a58aafd8a01c46"
server_key_version: "encryptionkey@0"
I20260927 18:44:31.205053 31965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:44:31.208487 31980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.209857 31965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:31.210023 31965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "c70d7e500113446787fbf4ea3b073ebd"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-rfd5"
server_key: "8022d1fa7bf0ebdc121a883bdf9ee0e4"
server_key_iv: "e074c3b802c0f9ff21a58aafd8a01c46"
server_key_version: "encryptionkey@0"
I20260927 18:44:31.210219 31965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:31.243268 31965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:31.247573 31965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:31.247879 31965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:31.248852 31965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:31.250258 31965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:31.250358 31965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.250511 31965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:31.250578 31965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.286887 31965 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:33407
I20260927 18:44:31.286978 32092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:33407 every 8 connection(s)
I20260927 18:44:31.288255 31965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:31.297458 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 31965
I20260927 18:44:31.297689 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:31.298307 31851 external_mini_cluster.cc:1428] Setting key aa08fbd051dac1f63830a211f5b4cace
Sep 27 18:44:31 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:31.309535 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:31.291442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:58811 (local address 127.31.26.254:44079)
0927 18:44:31.291921 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:31.291926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:31.292936 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 18:44:31.294893 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:31.294898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:31.295005 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:31.295120 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:31.296657 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.297714 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.299383 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.299848 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.300006 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:31.303856 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:31.303861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:31.303864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:31.303897 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:31.306514 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:31.307254 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:31.307261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:31.307269 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:31.307379 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:31.307794 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:31.307801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:31.307804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:31.308064 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:31.308215 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:31.308912 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 18:44:31.309163 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260927 18:44:31.310636 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:31.292052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44079 (local address 127.31.26.193:58811)
0927 18:44:31.292764 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:31.292805 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:31.294607 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:31.295373 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:31.295381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:31.295902 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:31.296486 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:31.296502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.297889 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.297894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:31.299231 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:31.299244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.300111 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.300119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:31.300373 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:31.301170 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:31.301208 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:31.303614 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:31.306663 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:31.306672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:31.306691 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:31.307116 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:31.307489 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:31.307494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:31.307498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:31.307686 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:31.308197 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:31.308205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:31.308552 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 18:44:31.308830 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 18:44:31.309418 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":292,"threads_started":1}
I20260927 18:44:31.312247 32093 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44079
I20260927 18:44:31.312547 32093 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:31.313333 32093 heartbeater.cc:511] Master 127.31.26.254:44079 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:44:31.315837 31903 ts_manager.cc:194] Registered new tserver with Master: c70d7e500113446787fbf4ea3b073ebd (127.31.26.193:33407)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:44:31.317512 31903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:58811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:44:31.336558 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44079
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:31.677775 32099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:31.678081 32099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:31.678241 32099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:31.685655 32099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:31.685796 32099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:31.685966 32099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:44:31.695916 32099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:31.697301 32099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:31.698704 32099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:31.711959 32105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:31.711933 32107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:31.712167 32104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:31.712882 32099 server_base.cc:1034] running on GCE node
I20260927 18:44:31.713537 32099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:31.714551 32099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:31.715786 32099 hybrid_clock.cc:648] HybridClock initialized: now 1790534671715731 us; error 62 us; skew 500 ppm
I20260927 18:44:31.716110 32099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:31 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:31.719280 32099 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:44:31.720839 32099 webserver.cc:460] Webserver started at http://127.31.26.194:43413/ using document root <none> and password file <none>
I20260927 18:44:31.721633 32099 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:31.721776 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:31.722096 32099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:31.725068 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "c35f5629f1f3448d8dca94ac6262ba0a"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-rfd5"
server_key: "95bf820b80d23f225cd2a4ef9c9a4f1e"
server_key_iv: "ee83397513b8e865a50a5a561d425bb1"
server_key_version: "encryptionkey@0"
I20260927 18:44:31.725919 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "c35f5629f1f3448d8dca94ac6262ba0a"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-rfd5"
server_key: "95bf820b80d23f225cd2a4ef9c9a4f1e"
server_key_iv: "ee83397513b8e865a50a5a561d425bb1"
server_key_version: "encryptionkey@0"
I20260927 18:44:31.731383 32099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:44:31.735662 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.737239 32099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:44:31.737434 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "c35f5629f1f3448d8dca94ac6262ba0a"
format_stamp: "Formatted at 2026-09-27 18:44:31 on dist-test-slave-rfd5"
server_key: "95bf820b80d23f225cd2a4ef9c9a4f1e"
server_key_iv: "ee83397513b8e865a50a5a561d425bb1"
server_key_version: "encryptionkey@0"
I20260927 18:44:31.737670 32099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:31.756932 32099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:31.761377 32099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:31.761669 32099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:31.762719 32099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:31.764001 32099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:31.764074 32099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.764168 32099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:31.764209 32099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:31.799031 32099 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:34355
I20260927 18:44:31.799067 32226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:34355 every 8 connection(s)
I20260927 18:44:31.800367 32099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:31.802770 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 32099
I20260927 18:44:31.802968 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:31.803444 31851 external_mini_cluster.cc:1428] Setting key bf95a821aaf8150876f88ec5b6b06534
Sep 27 18:44:31 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534671, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:44:31.822628 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:31.802942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:47527 (local address 127.31.26.254:44079)
0927 18:44:31.803163 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:31.803169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:31.804493 (+ 1324us) server_negotiation.cc:407] Connection header received
0927 18:44:31.806379 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:31.806388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:31.806479 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:31.806595 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:31.808092 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.809209 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.811783 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.812184 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.812280 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:31.816802 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:31.816807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:31.816810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:31.816843 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:31.819410 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:31.820162 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:31.820168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:31.820172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:31.820294 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:31.820788 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:31.820796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:31.820799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:31.821072 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:31.821274 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:31.822049 (+ 775us) server_negotiation.cc:299] Negotiation successful
0927 18:44:31.822285 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:44:31.823668 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:31.803469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44079 (local address 127.31.26.194:47527)
0927 18:44:31.804324 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:31.804376 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:31.806101 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:31.806810 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:31.806819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:31.807328 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:31.807923 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:31.807942 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.809364 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.809371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:31.811644 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:31.811661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:31.812403 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:31.812413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:31.812749 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:31.813586 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:31.813625 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:31.816537 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:31.819558 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:31.819568 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:31.819586 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:31.820036 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:31.820404 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:31.820414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:31.820418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:31.820681 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:31.821249 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:31.821256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:31.821700 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 18:44:31.821975 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 18:44:31.822509 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":341,"threads_started":1}
I20260927 18:44:31.825116 32227 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44079
I20260927 18:44:31.825470 32227 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:31.826318 32227 heartbeater.cc:511] Master 127.31.26.254:44079 requested a full tablet report, sending...
I20260927 18:44:31.827800 31903 ts_manager.cc:194] Registered new tserver with Master: c35f5629f1f3448d8dca94ac6262ba0a (127.31.26.194:34355)
I20260927 18:44:31.828608 31903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:47527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:44:31.843809 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44079
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:32.182786 32232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:32.183102 32232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:32.183261 32232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:32.190742 32232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:32.190883 32232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:32.191053 32232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:44:32.201463 32232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:32.202808 32232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:32.204144 32232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:32.218152 32238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:32.218210 32237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:32.218827 32240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:32.219280 32232 server_base.cc:1034] running on GCE node
I20260927 18:44:32.219949 32232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:32.220961 32232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:32.222208 32232 hybrid_clock.cc:648] HybridClock initialized: now 1790534672222136 us; error 67 us; skew 500 ppm
I20260927 18:44:32.222579 32232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:32.225919 32232 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:44:32.227537 32232 webserver.cc:460] Webserver started at http://127.31.26.195:46061/ using document root <none> and password file <none>
I20260927 18:44:32.228264 32232 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:32.228408 32232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:32.228797 32232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:32.231814 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "cdb04d473f63415a85decb6f7a2c2cf5"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-rfd5"
server_key: "e42cbf33e22e770936addfe8d0d27e9b"
server_key_iv: "8d2fc800c5b4a53005cbf22a30e02536"
server_key_version: "encryptionkey@0"
I20260927 18:44:32.232554 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "cdb04d473f63415a85decb6f7a2c2cf5"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-rfd5"
server_key: "e42cbf33e22e770936addfe8d0d27e9b"
server_key_iv: "8d2fc800c5b4a53005cbf22a30e02536"
server_key_version: "encryptionkey@0"
I20260927 18:44:32.238159 32232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:44:32.241715 32247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:32.243286 32232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:32.243464 32232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "cdb04d473f63415a85decb6f7a2c2cf5"
format_stamp: "Formatted at 2026-09-27 18:44:32 on dist-test-slave-rfd5"
server_key: "e42cbf33e22e770936addfe8d0d27e9b"
server_key_iv: "8d2fc800c5b4a53005cbf22a30e02536"
server_key_version: "encryptionkey@0"
I20260927 18:44:32.243678 32232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:32.266508 32232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:32.271081 32232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:32.271384 32232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:32.272363 32232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:32.273698 32232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:32.273801 32232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:32.274348 32232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:32.274425 32232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 18:44:32.310068 32232 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:43193
I20260927 18:44:32.310114 32359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:43193 every 8 connection(s)
I20260927 18:44:32.311475 32232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:32.321002 32093 heartbeater.cc:503] Master 127.31.26.254:44079 was elected leader, sending a full tablet report...
I20260927 18:44:32.321625 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 32232
I20260927 18:44:32.321830 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:32.322365 31851 external_mini_cluster.cc:1428] Setting key ce069519c8045d231c87f5c2faf854b1
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:32.331780 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.314102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:37279 (local address 127.31.26.254:44079)
0927 18:44:32.314320 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.314325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.315587 (+ 1262us) server_negotiation.cc:407] Connection header received
0927 18:44:32.317406 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.317410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.317505 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.317597 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.319460 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.320199 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.322037 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.322477 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.322600 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.326772 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.326777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.326781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.326814 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.329222 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.329865 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.329869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.329871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.329985 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.330361 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.330363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.330365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.330611 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.330730 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.331366 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.331525 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:44:32.332742 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.314649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44079 (local address 127.31.26.195:37279)
0927 18:44:32.315425 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:32.315464 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:32.317146 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:32.317809 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:32.317817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:32.318742 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:32.319311 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:32.319329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.320324 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.320328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:32.321906 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:32.321922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.322733 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.322742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:32.323056 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.323815 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:32.323838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:32.326571 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:32.329337 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:32.329344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:32.329357 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:32.329759 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:32.330068 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:32.330072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:32.330074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:32.330269 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:32.330719 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:32.330725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:32.331031 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 18:44:32.331274 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 18:44:32.331655 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":293,"threads_started":1}
I20260927 18:44:32.334086 32360 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44079
I20260927 18:44:32.334380 32360 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:32.335031 32360 heartbeater.cc:511] Master 127.31.26.254:44079 requested a full tablet report, sending...
I20260927 18:44:32.336351 31903 ts_manager.cc:194] Registered new tserver with Master: cdb04d473f63415a85decb6f7a2c2cf5 (127.31.26.195:43193)
I20260927 18:44:32.337302 31903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:37279
I20260927 18:44:32.338897 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:32.382743 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.367383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.31.26.193:33407)
0927 18:44:32.368025 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.368035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.368129 (+ 94us) server_negotiation.cc:407] Connection header received
0927 18:44:32.368350 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.368358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.368590 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.368759 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.370254 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.371245 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.372508 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.373076 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.373266 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.377294 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.377305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.377320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.377361 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.379940 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.380600 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.380614 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.380616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.380751 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.381191 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.381199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.381202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.381534 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.381847 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.382129 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.382389 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":293,"threads_started":1}
W20260927 18:44:32.384040 32008 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:32.428169 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.415087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.31.26.193:33407)
0927 18:44:32.415337 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.415346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.415518 (+ 172us) server_negotiation.cc:407] Connection header received
0927 18:44:32.415725 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.415729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.415826 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.415914 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.417273 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.417962 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.419118 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.419531 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.419623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.423429 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.423434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.423437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.423471 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.425948 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.426559 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.426562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.426564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.426689 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.426986 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.426989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.426991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.427225 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.427394 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.427623 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.427837 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:44:32.429666 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 31965
I20260927 18:44:32.442557 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.430648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.31.26.193:33407)
0927 18:44:32.430797 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.430802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.431085 (+ 283us) server_negotiation.cc:407] Connection header received
0927 18:44:32.431220 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.431224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.431276 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.431347 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.432453 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.433184 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.434468 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.434883 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.434974 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.437616 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.437627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.437630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.437662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.440010 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.440641 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.440645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.440648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.440771 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.441188 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.441192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.441195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.441533 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.441724 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.441991 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.442206 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:44:32.443353 32088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:32.526271 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 31965
I20260927 18:44:32.542088 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 32099
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:44:32.558228 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.542983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.31.26.194:34355)
0927 18:44:32.543507 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.543514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.543540 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:44:32.543603 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.543609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.543824 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.544002 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.545293 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.546305 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.547615 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.548202 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.548348 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.552481 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.552493 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.552509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.552688 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.555668 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.556291 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.556298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.556300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.556415 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.556764 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.556767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.556769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.557268 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.557493 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.557751 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.557941 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
I20260927 18:44:32.559485 32222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:32.645278 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 32099
I20260927 18:44:32.660938 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 32232
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[31858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:44:32.677582 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:32.662010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.31.26.195:43193)
0927 18:44:32.662641 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:32.662649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:32.662679 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:44:32.662737 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:32.662743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:32.662958 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:32.663128 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:32.664422 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.665559 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.666849 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:32.667435 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:32.667576 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:32.671977 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:32.671991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:32.672008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:32.672052 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:32.674974 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.675677 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.675684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.675686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.675806 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:32.676101 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:32.676104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:32.676106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:32.676440 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:32.676666 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:32.676917 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 18:44:32.677196 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":344,"threads_started":1}
I20260927 18:44:32.678892 32355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:32.762974 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 32232
I20260927 18:44:32.778919 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 31871
I20260927 18:44:32.780208 31933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:32.882992 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 31871
2026-09-27T18:44:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3949 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[32381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[32381](info): set up 2 sockets
Sep 27 18:44:32 dist-test-slave-rfd5 krb5kdc[32381](info): commencing operation
krb5kdc: starting...
W20260927 18:44:33.969023 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:34 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:34.145236 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34463
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:34463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:34.472653 32394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:34.472955 32394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:34.473045 32394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:34.480398 32394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:34.480531 32394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:34.480631 32394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:34.480726 32394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:34.480774 32394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:34.491166 32394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:34463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:34.492487 32394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:34.493817 32394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:34.506141 32400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:34.506422 32402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:34.506259 32399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:34.507287 32394 server_base.cc:1034] running on GCE node
I20260927 18:44:34.508168 32394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:34.509819 32394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:34.511059 32394 hybrid_clock.cc:648] HybridClock initialized: now 1790534674510997 us; error 59 us; skew 500 ppm
I20260927 18:44:34.511471 32394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:34 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:34.514765 32394 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:34.516444 32394 webserver.cc:460] Webserver started at http://127.31.26.254:43667/ using document root <none> and password file <none>
I20260927 18:44:34.517254 32394 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:34.517405 32394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:34.517735 32394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:34.520763 32394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "f99317611af041f78504bcc3e9bf69a8"
format_stamp: "Formatted at 2026-09-27 18:44:34 on dist-test-slave-rfd5"
server_key: "e8453b8a0273452aae534b4aa397dfde"
server_key_iv: "add3eeee20b78b5438878a3f0cd4e338"
server_key_version: "encryptionkey@0"
I20260927 18:44:34.521559 32394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "f99317611af041f78504bcc3e9bf69a8"
format_stamp: "Formatted at 2026-09-27 18:44:34 on dist-test-slave-rfd5"
server_key: "e8453b8a0273452aae534b4aa397dfde"
server_key_iv: "add3eeee20b78b5438878a3f0cd4e338"
server_key_version: "encryptionkey@0"
I20260927 18:44:34.526955 32394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:34.530649 32409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:34.532115 32394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 18:44:34.532313 32394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "f99317611af041f78504bcc3e9bf69a8"
format_stamp: "Formatted at 2026-09-27 18:44:34 on dist-test-slave-rfd5"
server_key: "e8453b8a0273452aae534b4aa397dfde"
server_key_iv: "add3eeee20b78b5438878a3f0cd4e338"
server_key_version: "encryptionkey@0"
I20260927 18:44:34.532519 32394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:34.548532 32394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:34.552995 32394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:34.553463 32394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:34.572485 32394 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:34463
I20260927 18:44:34.572469 32460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:34463 every 8 connection(s)
I20260927 18:44:34.574195 32394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:34.578202 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:34.582873 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 32394
I20260927 18:44:34.583142 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:34.583814 31851 external_mini_cluster.cc:1428] Setting key c26f11a028596f008479616089bdf5f4
I20260927 18:44:34.590744 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8: Bootstrap starting.
I20260927 18:44:34.594643 32461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:34.595757 32461 log.cc:824] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:34.599480 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8: No bootstrap required, opened a new log
Sep 27 18:44:34 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:34.610284 32461 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99317611af041f78504bcc3e9bf69a8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34463 } }
I20260927 18:44:34.610746 32461 raft_consensus.cc:374] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:34.610895 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:34.585979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.31.26.254:34463)
0927 18:44:34.586885 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:34.586901 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:34.586960 (+ 59us) server_negotiation.cc:407] Connection header received
0927 18:44:34.589492 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:34.589524 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:34.590055 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:34.590537 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:34.592046 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.594221 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.595588 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:34.596637 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:34.596779 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:34.601797 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:34.601813 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:34.601838 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:34.601893 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:34.606040 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.606768 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.606783 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.606792 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.607017 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:34.607386 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:34.607391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:34.607394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:34.608197 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:34.608499 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:34.608932 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 18:44:34.609455 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":389,"threads_started":1}
I20260927 18:44:34.610860 32461 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99317611af041f78504bcc3e9bf69a8, State: Initialized, Role: FOLLOWER
I20260927 18:44:34.611593 32461 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99317611af041f78504bcc3e9bf69a8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34463 } }
I20260927 18:44:34.611879 32461 raft_consensus.cc:388] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:34.612015 32461 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:34.612205 32461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:34.614801 32461 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99317611af041f78504bcc3e9bf69a8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34463 } }
I20260927 18:44:34.615296 32461 leader_election.cc:302] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f99317611af041f78504bcc3e9bf69a8; no voters:
I20260927 18:44:34.615880 32461 leader_election.cc:288] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:34.616058 32466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:34.616760 32466 raft_consensus.cc:686] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [term 1 LEADER]: Becoming Leader. State: Replica: f99317611af041f78504bcc3e9bf69a8, State: Running, Role: LEADER
I20260927 18:44:34.617313 32466 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99317611af041f78504bcc3e9bf69a8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34463 } }
I20260927 18:44:34.618021 32461 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:34.620158 32467 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f99317611af041f78504bcc3e9bf69a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99317611af041f78504bcc3e9bf69a8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34463 } } }
I20260927 18:44:34.620635 32467 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:34.620901 32468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99317611af041f78504bcc3e9bf69a8. Latest consensus state: current_term: 1 leader_uuid: "f99317611af041f78504bcc3e9bf69a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99317611af041f78504bcc3e9bf69a8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34463 } } }
I20260927 18:44:34.621266 32468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:34.623591 32475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:34.628051 32475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:34.637162 32475 catalog_manager.cc:1347] Generated new cluster ID: b72047df88b648b38a7d1c0c52d3ce1c
I20260927 18:44:34.637295 32475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:34.653277 32475 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:34.655248 32475 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:34.667789 32475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f99317611af041f78504bcc3e9bf69a8: Generated new TSK 0
I20260927 18:44:34.668789 32475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:44:34.717049 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34463
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:35.056042 32487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:35.056352 32487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:35.056449 32487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:35.064338 32487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:35.064459 32487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:35.064575 32487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:44:35.074702 32487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:35.076025 32487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:35.077457 32487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:35.091022 32492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:35.091084 32493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:35.091001 32495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:35.092190 32487 server_base.cc:1034] running on GCE node
I20260927 18:44:35.092828 32487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:35.093978 32487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:35.095237 32487 hybrid_clock.cc:648] HybridClock initialized: now 1790534675095188 us; error 54 us; skew 500 ppm
I20260927 18:44:35.095597 32487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:35 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534675, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:35.099289 32487 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:44:35.101058 32487 webserver.cc:460] Webserver started at http://127.31.26.193:41895/ using document root <none> and password file <none>
I20260927 18:44:35.101884 32487 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:35.102046 32487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:35.102381 32487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:35.105671 32487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "6d8f56dff30241bc97075e245cbdb154"
format_stamp: "Formatted at 2026-09-27 18:44:35 on dist-test-slave-rfd5"
server_key: "2112e534d8d457a94c37bc561a0c64fc"
server_key_iv: "db1c0f64f71b10b89c73b7c29b58c8e9"
server_key_version: "encryptionkey@0"
I20260927 18:44:35.106513 32487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "6d8f56dff30241bc97075e245cbdb154"
format_stamp: "Formatted at 2026-09-27 18:44:35 on dist-test-slave-rfd5"
server_key: "2112e534d8d457a94c37bc561a0c64fc"
server_key_iv: "db1c0f64f71b10b89c73b7c29b58c8e9"
server_key_version: "encryptionkey@0"
I20260927 18:44:35.112325 32487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 18:44:35.115903 32502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:35.117384 32487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:44:35.117563 32487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "6d8f56dff30241bc97075e245cbdb154"
format_stamp: "Formatted at 2026-09-27 18:44:35 on dist-test-slave-rfd5"
server_key: "2112e534d8d457a94c37bc561a0c64fc"
server_key_iv: "db1c0f64f71b10b89c73b7c29b58c8e9"
server_key_version: "encryptionkey@0"
I20260927 18:44:35.117782 32487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:35.131063 32487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:35.135190 32487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:35.135499 32487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:35.136512 32487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:35.137930 32487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:35.138037 32487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:35.138185 32487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:35.138257 32487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:35.171271 32487 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:37053
I20260927 18:44:35.171330 32614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:37053 every 8 connection(s)
I20260927 18:44:35.172585 32487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:35.175491 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 32487
I20260927 18:44:35.175750 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:35.176401 31851 external_mini_cluster.cc:1428] Setting key 0b38cf1ef2fe7d83661d967c30264ed6
Sep 27 18:44:35 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534675, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:44:35.195530 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.175343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:42987 (local address 127.31.26.254:34463)
0927 18:44:35.175899 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.175906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.177207 (+ 1301us) server_negotiation.cc:407] Connection header received
0927 18:44:35.179132 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.179139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.179233 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.179373 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:35.180843 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.182128 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.183798 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.184346 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.184488 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.189171 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:35.189177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:35.189181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:35.189215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:35.191978 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:35.192986 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:35.192991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:35.192993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:35.193210 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:35.193764 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:35.193768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:35.193771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:35.194127 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:35.194265 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.195078 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.195259 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:44:35.196640 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.175968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34463 (local address 127.31.26.193:42987)
0927 18:44:35.176807 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.176854 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.178846 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.179557 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.179565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.180076 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:35.180651 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.180672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.182305 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.182309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.183573 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.183588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.184588 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.184598 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.184939 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.185730 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:35.185764 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:35.188862 (+ 3098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:35.192146 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:35.192165 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:35.192186 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:35.192827 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:35.193342 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:35.193349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:35.193356 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:35.193623 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:35.194287 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:35.194295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:35.194641 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.194961 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.195385 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":334,"threads_started":1}
I20260927 18:44:35.198328 32615 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34463
I20260927 18:44:35.198738 32615 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:35.199694 32615 heartbeater.cc:511] Master 127.31.26.254:34463 requested a full tablet report, sending...
I20260927 18:44:35.202270 32426 ts_manager.cc:194] Registered new tserver with Master: 6d8f56dff30241bc97075e245cbdb154 (127.31.26.193:37053)
I20260927 18:44:35.203747 32426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:42987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:44:35.220834 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34463
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:35.551141 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:35.551442 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:35.551541 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:35.558980 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:35.559115 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:35.559233 32621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:44:35.569319 32621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:35.570637 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:35.572129 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:35.585831 32626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:35.585868 32627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:35.586520 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:35.588085 32621 server_base.cc:1034] running on GCE node
I20260927 18:44:35.588771 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:35.589923 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:35.591182 32621 hybrid_clock.cc:648] HybridClock initialized: now 1790534675591116 us; error 60 us; skew 500 ppm
I20260927 18:44:35.591531 32621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:35 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534675, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:35.595018 32621 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:44:35.596609 32621 webserver.cc:460] Webserver started at http://127.31.26.194:41223/ using document root <none> and password file <none>
I20260927 18:44:35.597416 32621 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:35.597558 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:35.597878 32621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:35.600864 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "4db8d0513ea6477fbcfec952991ec7f2"
format_stamp: "Formatted at 2026-09-27 18:44:35 on dist-test-slave-rfd5"
server_key: "5784dc952e311b56538e19449aea290c"
server_key_iv: "bb879eb5158601e948aa1451d411aec9"
server_key_version: "encryptionkey@0"
I20260927 18:44:35.601761 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "4db8d0513ea6477fbcfec952991ec7f2"
format_stamp: "Formatted at 2026-09-27 18:44:35 on dist-test-slave-rfd5"
server_key: "5784dc952e311b56538e19449aea290c"
server_key_iv: "bb879eb5158601e948aa1451d411aec9"
server_key_version: "encryptionkey@0"
I20260927 18:44:35.607296 32621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:44:35.611268 32636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:35.612726 32621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:44:35.612913 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "4db8d0513ea6477fbcfec952991ec7f2"
format_stamp: "Formatted at 2026-09-27 18:44:35 on dist-test-slave-rfd5"
server_key: "5784dc952e311b56538e19449aea290c"
server_key_iv: "bb879eb5158601e948aa1451d411aec9"
server_key_version: "encryptionkey@0"
I20260927 18:44:35.613191 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:35.634979 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:35.639719 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:35.639964 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:35.640911 32621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:35.642309 32621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:35.642391 32621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:35.642477 32621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:35.642520 32621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:35.675863 32621 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:45197
I20260927 18:44:35.675910 32748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:45197 every 8 connection(s)
I20260927 18:44:35.677556 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:35.679358 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 32621
I20260927 18:44:35.679541 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:35.680009 31851 external_mini_cluster.cc:1428] Setting key 7daef6bf041b317c79a4336eb0c00326
Sep 27 18:44:35 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534675, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:44:35.699544 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.680248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:45897 (local address 127.31.26.254:34463)
0927 18:44:35.680460 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:35.680467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:35.681868 (+ 1401us) server_negotiation.cc:407] Connection header received
0927 18:44:35.683775 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:35.683781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:35.683881 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:35.683993 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:35.685639 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.686331 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.687927 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.688361 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.688523 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.692562 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:35.692572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:35.692581 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:35.692633 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:35.696329 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:35.697241 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:35.697249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:35.697255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:35.697392 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:35.697837 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:35.697841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:35.697843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:35.698099 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:35.698209 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:35.699122 (+ 913us) server_negotiation.cc:299] Negotiation successful
0927 18:44:35.699306 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:44:35.701439 32751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:35.680850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34463 (local address 127.31.26.194:45897)
0927 18:44:35.681699 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:35.681756 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:35.683486 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:35.684245 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:35.684253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:35.684781 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:35.685458 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.685478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.686487 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.686495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.687774 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:35.687786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:35.688620 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:35.688627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:35.688862 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:35.689667 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:35.689698 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:35.692253 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:35.696537 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:35.696546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:35.696561 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:35.697026 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:35.697521 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:35.697526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:35.697529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:35.697732 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:35.698248 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:35.698255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:35.698657 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 18:44:35.698971 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 18:44:35.699568 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260927 18:44:35.703192 32749 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34463
I20260927 18:44:35.703565 32749 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:35.704320 32749 heartbeater.cc:511] Master 127.31.26.254:34463 requested a full tablet report, sending...
I20260927 18:44:35.706049 32426 ts_manager.cc:194] Registered new tserver with Master: 4db8d0513ea6477fbcfec952991ec7f2 (127.31.26.194:45197)
I20260927 18:44:35.707345 32426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:45897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:44:35.722043 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34463
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:36.050042 32754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:36.050338 32754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:36.050437 32754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:36.058539 32754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:36.058661 32754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:36.058779 32754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:44:36.068792 32754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:36.070156 32754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:36.071434 32754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:36.085129 32759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:36.085071 32760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:36.085158 32762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:36.085840 32754 server_base.cc:1034] running on GCE node
I20260927 18:44:36.086474 32754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:36.087450 32754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:36.088701 32754 hybrid_clock.cc:648] HybridClock initialized: now 1790534676088610 us; error 86 us; skew 500 ppm
I20260927 18:44:36.089007 32754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:36.092331 32754 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:44:36.093914 32754 webserver.cc:460] Webserver started at http://127.31.26.195:45183/ using document root <none> and password file <none>
I20260927 18:44:36.094610 32754 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:36.094715 32754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:36.094964 32754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:36.097951 32754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "235c556e83334f7289cb65af33a36308"
format_stamp: "Formatted at 2026-09-27 18:44:36 on dist-test-slave-rfd5"
server_key: "2bfce846eea899cd4cbeb8ebe3d2c4a2"
server_key_iv: "5990d25dfbc2da230e2f4edd363d85ff"
server_key_version: "encryptionkey@0"
I20260927 18:44:36.098778 32754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "235c556e83334f7289cb65af33a36308"
format_stamp: "Formatted at 2026-09-27 18:44:36 on dist-test-slave-rfd5"
server_key: "2bfce846eea899cd4cbeb8ebe3d2c4a2"
server_key_iv: "5990d25dfbc2da230e2f4edd363d85ff"
server_key_version: "encryptionkey@0"
I20260927 18:44:36.104365 32754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:44:36.108258 301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:36.109886 32754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:36.110113 32754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "235c556e83334f7289cb65af33a36308"
format_stamp: "Formatted at 2026-09-27 18:44:36 on dist-test-slave-rfd5"
server_key: "2bfce846eea899cd4cbeb8ebe3d2c4a2"
server_key_iv: "5990d25dfbc2da230e2f4edd363d85ff"
server_key_version: "encryptionkey@0"
I20260927 18:44:36.110425 32754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:36.125149 32754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:36.129359 32754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:36.129673 32754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:36.130674 32754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:36.132030 32754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:36.132112 32754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:36.132285 32754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:36.132335 32754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:36.165387 32754 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:46507
I20260927 18:44:36.165426 413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:46507 every 8 connection(s)
I20260927 18:44:36.166965 32754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:36.169513 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 32754
I20260927 18:44:36.169735 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:36.170168 31851 external_mini_cluster.cc:1428] Setting key 01d6c26cc482b3e7669492c1c9f8ee88
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:36.189065 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.169733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:34543 (local address 127.31.26.254:34463)
0927 18:44:36.169937 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.169944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.172088 (+ 2144us) server_negotiation.cc:407] Connection header received
0927 18:44:36.174988 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.174994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.175080 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.175201 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.176724 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.177564 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.179144 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.179592 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.179692 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.183206 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.183211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.183214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.183246 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.186339 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.187073 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.187077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.187079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.187197 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.187564 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.187567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.187570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.187821 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.187926 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.188651 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.188821 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:44:36.190119 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.170819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34463 (local address 127.31.26.195:34543)
0927 18:44:36.171919 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:36.171972 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:36.174697 (+ 2725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:36.175413 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:36.175422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:36.175962 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:36.176549 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.176567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.177718 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.177722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.178970 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.178982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.179785 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.179789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.179994 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.180727 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:36.180750 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:36.182998 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:36.186468 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:36.186474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:36.186489 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:36.186973 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:36.187287 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:36.187290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:36.187293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:36.187476 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:36.187929 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:36.187935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:36.188269 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 18:44:36.188511 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 18:44:36.188927 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":506,"threads_started":1}
I20260927 18:44:36.191735 414 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34463
I20260927 18:44:36.192058 414 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:36.193300 414 heartbeater.cc:511] Master 127.31.26.254:34463 requested a full tablet report, sending...
I20260927 18:44:36.194800 32426 ts_manager.cc:194] Registered new tserver with Master: 235c556e83334f7289cb65af33a36308 (127.31.26.195:46507)
I20260927 18:44:36.195611 32426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:34543
I20260927 18:44:36.198526 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:44:36.206856 32615 heartbeater.cc:503] Master 127.31.26.254:34463 was elected leader, sending a full tablet report...
I20260927 18:44:36.217825 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.204795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.31.26.254:34463)
0927 18:44:36.205044 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.205050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.205422 (+ 372us) server_negotiation.cc:407] Connection header received
0927 18:44:36.205563 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.205566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.205638 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.205723 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.207230 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.208108 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.209622 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.210060 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.210167 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.212596 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.212602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.212605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.212633 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.215157 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.215786 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.215790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.215793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.215912 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.216237 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.216240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.216242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.216538 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.216822 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.217203 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.217421 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:44:36.224491 32426 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:36.231768 32426 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:36.276499 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.244689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:46507 (local address 127.0.0.1:49694)
0927 18:44:36.246261 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:36.246302 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:36.246460 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:36.267163 (+ 20703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:36.267177 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:36.267216 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:36.267669 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.267686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.270661 (+ 2975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.270670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.273608 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.273625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.275307 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.275315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.275519 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.275580 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:44:36.275719 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:44:36.275935 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"mutex_wait_us":249,"thread_start_us":886,"threads_started":1}
I20260927 18:44:36.276646 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.246137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:45197 (local address 127.0.0.1:45124)
0927 18:44:36.264437 (+ 18300us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:36.264472 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:36.264709 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:36.268510 (+ 3801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:36.268520 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:36.268544 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:36.269370 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.269382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.270724 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.270729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.274144 (+ 3415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.274174 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.275657 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.275664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.275839 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.275880 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:44:36.275965 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:44:36.276114 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18074,"mutex_wait_us":253,"thread_start_us":18711,"threads_started":1}
I20260927 18:44:36.277415 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.246973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.31.26.194:45197)
0927 18:44:36.267910 (+ 20937us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.267920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.267947 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:44:36.268044 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.268051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.268296 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.268492 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:36.269534 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.270557 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.274354 (+ 3797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.275396 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.275524 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.275774 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.276883 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.277066 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20746,"thread_start_us":20608,"threads_started":1}
I20260927 18:44:36.277606 427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.245031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.31.26.195:46507)
0927 18:44:36.266292 (+ 21261us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.266303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.266349 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:44:36.266478 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.266486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.266814 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.267927 (+ 1113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:36.268764 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.270511 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.273771 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.275094 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.275235 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.275479 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.277038 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.277247 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21067,"thread_start_us":20892,"threads_started":1}
I20260927 18:44:36.278805 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.246705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37053 (local address 127.0.0.1:40884)
0927 18:44:36.267186 (+ 20481us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:36.267206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:36.267344 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:36.268164 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:36.268169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:36.268197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:36.268670 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.268682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.270721 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.270729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.274144 (+ 3415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.274174 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.275975 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.275983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.276166 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.276203 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:44:36.276290 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:44:36.276437 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20335,"mutex_wait_us":219,"thread_start_us":20195,"threads_started":1}
I20260927 18:44:36.279167 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.247123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.31.26.193:37053)
0927 18:44:36.266716 (+ 19593us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.266724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.267336 (+ 612us) server_negotiation.cc:407] Connection header received
0927 18:44:36.267519 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.267533 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.267836 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.268055 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:36.268878 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.270511 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.274331 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.275766 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.275904 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.276148 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.277137 (+ 989us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.277288 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19404,"thread_start_us":19277,"threads_started":1}
I20260927 18:44:36.280882 32684 tablet_service.cc:1467] Processing CreateTablet for tablet b3b2efdd21cd4dbdb11f964551a38967 (DEFAULT_TABLE table=test-table [id=b06655a5a5a844ad8f0c9a8913b59aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:36.280880 349 tablet_service.cc:1467] Processing CreateTablet for tablet b3b2efdd21cd4dbdb11f964551a38967 (DEFAULT_TABLE table=test-table [id=b06655a5a5a844ad8f0c9a8913b59aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:36.282358 32684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b2efdd21cd4dbdb11f964551a38967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:36.282368 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b2efdd21cd4dbdb11f964551a38967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:36.286260 32550 tablet_service.cc:1467] Processing CreateTablet for tablet b3b2efdd21cd4dbdb11f964551a38967 (DEFAULT_TABLE table=test-table [id=b06655a5a5a844ad8f0c9a8913b59aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:36.287879 32550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b2efdd21cd4dbdb11f964551a38967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:36.293974 432 tablet_bootstrap.cc:492] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: Bootstrap starting.
I20260927 18:44:36.297616 433 tablet_bootstrap.cc:492] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: Bootstrap starting.
I20260927 18:44:36.298429 432 tablet_bootstrap.cc:654] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:36.299762 432 log.cc:824] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:36.301666 433 tablet_bootstrap.cc:654] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:36.302935 433 log.cc:824] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:36.303222 435 tablet_bootstrap.cc:492] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: Bootstrap starting.
I20260927 18:44:36.306058 432 tablet_bootstrap.cc:492] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: No bootstrap required, opened a new log
I20260927 18:44:36.306438 432 ts_tablet_manager.cc:1397] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260927 18:44:36.306465 435 tablet_bootstrap.cc:654] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:36.307456 435 log.cc:824] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:36.310818 435 tablet_bootstrap.cc:492] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: No bootstrap required, opened a new log
I20260927 18:44:36.311144 435 ts_tablet_manager.cc:1397] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 18:44:36.316373 433 tablet_bootstrap.cc:492] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: No bootstrap required, opened a new log
I20260927 18:44:36.316711 433 ts_tablet_manager.cc:1397] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260927 18:44:36.320003 435 raft_consensus.cc:348] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.320416 435 raft_consensus.cc:374] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:36.320515 435 raft_consensus.cc:729] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8f56dff30241bc97075e245cbdb154, State: Initialized, Role: FOLLOWER
I20260927 18:44:36.320644 432 raft_consensus.cc:348] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.321151 432 raft_consensus.cc:374] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:36.321269 432 raft_consensus.cc:729] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235c556e83334f7289cb65af33a36308, State: Initialized, Role: FOLLOWER
I20260927 18:44:36.321354 435 consensus_queue.cc:260] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.321892 432 consensus_queue.cc:260] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.323314 414 heartbeater.cc:503] Master 127.31.26.254:34463 was elected leader, sending a full tablet report...
I20260927 18:44:36.323868 432 ts_tablet_manager.cc:1428] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260927 18:44:36.325562 435 ts_tablet_manager.cc:1428] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 18:44:36.327910 433 raft_consensus.cc:348] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.328413 433 raft_consensus.cc:374] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:36.328521 433 raft_consensus.cc:729] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db8d0513ea6477fbcfec952991ec7f2, State: Initialized, Role: FOLLOWER
I20260927 18:44:36.329340 433 consensus_queue.cc:260] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.330503 32749 heartbeater.cc:503] Master 127.31.26.254:34463 was elected leader, sending a full tablet report...
I20260927 18:44:36.331269 433 ts_tablet_manager.cc:1428] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
W20260927 18:44:36.418720 415 tablet.cc:2369] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:36.425426 32616 tablet.cc:2369] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:36.429538 32750 tablet.cc:2369] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:36.451915 438 raft_consensus.cc:482] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:36.452251 438 raft_consensus.cc:504] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.454130 438 leader_election.cc:288] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 235c556e83334f7289cb65af33a36308 (127.31.26.195:46507), 4db8d0513ea6477fbcfec952991ec7f2 (127.31.26.194:45197)
I20260927 18:44:36.461447 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.454828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:33417 (local address 127.31.26.194:45197)
0927 18:44:36.455021 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.455026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.455429 (+ 403us) server_negotiation.cc:407] Connection header received
0927 18:44:36.455671 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.455679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.455766 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.455844 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:36.456488 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.457480 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.459628 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.460378 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.460566 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.460654 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.461016 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.461165 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:44:36.461445 427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.454463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:56923 (local address 127.31.26.195:46507)
0927 18:44:36.454674 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.454684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.455599 (+ 915us) server_negotiation.cc:407] Connection header received
0927 18:44:36.455772 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.455775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.455859 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.455949 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:36.456611 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.457535 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.459575 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.460360 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.460526 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.460629 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.461018 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.461176 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:44:36.461619 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.454485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:46507 (local address 127.31.26.193:56923)
0927 18:44:36.455482 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:36.455496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:36.455648 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:36.456004 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:36.456006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:36.456038 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:36.456423 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.456436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.457699 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.457704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.459410 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.459425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.460605 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.460612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.460844 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.460874 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:44:36.460984 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:44:36.461163 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":48,"thread_start_us":703,"threads_started":1}
I20260927 18:44:36.461814 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.454463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:45197 (local address 127.31.26.193:33417)
0927 18:44:36.455284 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:36.455310 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:36.455497 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:36.455886 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:36.455893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:36.455924 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:36.456339 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.456351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.457615 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.457623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.459440 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:36.459451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.460652 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.460659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:36.460854 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.460878 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:44:36.460985 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:44:36.461142 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":135,"thread_start_us":413,"threads_started":1}
I20260927 18:44:36.463124 369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b2efdd21cd4dbdb11f964551a38967" candidate_uuid: "6d8f56dff30241bc97075e245cbdb154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235c556e83334f7289cb65af33a36308" is_pre_election: true
I20260927 18:44:36.463150 32704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b2efdd21cd4dbdb11f964551a38967" candidate_uuid: "6d8f56dff30241bc97075e245cbdb154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db8d0513ea6477fbcfec952991ec7f2" is_pre_election: true
I20260927 18:44:36.463621 32704 raft_consensus.cc:2454] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8f56dff30241bc97075e245cbdb154 in term 0.
I20260927 18:44:36.463624 369 raft_consensus.cc:2454] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8f56dff30241bc97075e245cbdb154 in term 0.
I20260927 18:44:36.464308 32503 leader_election.cc:302] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 235c556e83334f7289cb65af33a36308, 4db8d0513ea6477fbcfec952991ec7f2, 6d8f56dff30241bc97075e245cbdb154; no voters:
I20260927 18:44:36.464689 438 raft_consensus.cc:2781] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:36.464809 438 raft_consensus.cc:482] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:36.464941 438 raft_consensus.cc:3037] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:36.467118 438 raft_consensus.cc:504] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.467795 438 leader_election.cc:288] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [CANDIDATE]: Term 1 election: Requested vote from peers 235c556e83334f7289cb65af33a36308 (127.31.26.195:46507), 4db8d0513ea6477fbcfec952991ec7f2 (127.31.26.194:45197)
I20260927 18:44:36.468357 32704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b2efdd21cd4dbdb11f964551a38967" candidate_uuid: "6d8f56dff30241bc97075e245cbdb154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db8d0513ea6477fbcfec952991ec7f2"
I20260927 18:44:36.468374 369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b2efdd21cd4dbdb11f964551a38967" candidate_uuid: "6d8f56dff30241bc97075e245cbdb154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235c556e83334f7289cb65af33a36308"
I20260927 18:44:36.468542 32704 raft_consensus.cc:3037] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:36.468600 369 raft_consensus.cc:3037] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:36.470845 32704 raft_consensus.cc:2454] T b3b2efdd21cd4dbdb11f964551a38967 P 4db8d0513ea6477fbcfec952991ec7f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8f56dff30241bc97075e245cbdb154 in term 1.
I20260927 18:44:36.470851 369 raft_consensus.cc:2454] T b3b2efdd21cd4dbdb11f964551a38967 P 235c556e83334f7289cb65af33a36308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8f56dff30241bc97075e245cbdb154 in term 1.
I20260927 18:44:36.471375 32504 leader_election.cc:302] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db8d0513ea6477fbcfec952991ec7f2, 6d8f56dff30241bc97075e245cbdb154; no voters:
I20260927 18:44:36.471727 438 raft_consensus.cc:2781] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:36.472129 438 raft_consensus.cc:686] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [term 1 LEADER]: Becoming Leader. State: Replica: 6d8f56dff30241bc97075e245cbdb154, State: Running, Role: LEADER
I20260927 18:44:36.472607 438 consensus_queue.cc:237] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } }
I20260927 18:44:36.477836 32425 catalog_manager.cc:5495] T b3b2efdd21cd4dbdb11f964551a38967 P 6d8f56dff30241bc97075e245cbdb154 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d8f56dff30241bc97075e245cbdb154 (127.31.26.193). New cstate: current_term: 1 leader_uuid: "6d8f56dff30241bc97075e245cbdb154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235c556e83334f7289cb65af33a36308" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d8f56dff30241bc97075e245cbdb154" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db8d0513ea6477fbcfec952991ec7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45197 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:36.522204 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.506310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.31.26.193:37053)
0927 18:44:36.506510 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.506515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.507019 (+ 504us) server_negotiation.cc:407] Connection header received
0927 18:44:36.507247 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.507252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.507325 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.507432 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.509009 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.509795 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.511066 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.511482 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.511622 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.515929 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.515946 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.515963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.516000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.519556 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.520222 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.520229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.520231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.520381 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.520682 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.520685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.520687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.521003 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.521278 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.521615 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.521888 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:36.567312 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.553627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.31.26.193:37053)
0927 18:44:36.553896 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.553903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.554134 (+ 231us) server_negotiation.cc:407] Connection header received
0927 18:44:36.554305 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.554309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.554373 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.554462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.555830 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.556561 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.557840 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.558271 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.558365 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.562546 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.562551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.562555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.562593 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.565029 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.565656 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.565659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.565661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.565779 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.566131 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.566134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.566136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.566392 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.566594 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.566789 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.566956 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:36.614177 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.599432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.31.26.193:37053)
0927 18:44:36.599706 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.599714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.600058 (+ 344us) server_negotiation.cc:407] Connection header received
0927 18:44:36.600336 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.600340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.600418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.600530 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.602079 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.602827 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.604074 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.604484 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.604629 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.609263 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.609269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.609272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.609308 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.611827 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.612460 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.612464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.612466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.612603 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.612907 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.612910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.612912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.613260 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.613440 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.613726 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.613893 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 18:44:36.615969 32530 tablet_service.cc:2908] Rejecting scan request for tablet b3b2efdd21cd4dbdb11f964551a38967: Uninitialized: clean time has not yet been initialized
I20260927 18:44:36.619678 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 32487
I20260927 18:44:36.633224 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.620827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.31.26.193:37053)
0927 18:44:36.620968 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.620972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.621428 (+ 456us) server_negotiation.cc:407] Connection header received
0927 18:44:36.621578 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.621582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.621733 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.621862 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.623171 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.623892 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.625164 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.625568 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.625670 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.627945 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.627950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.627953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.627988 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.630749 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.631458 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.631462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.631464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.631613 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.631952 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.631955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.631958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.632237 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.632413 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.632693 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.632861 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 18:44:36.633910 32610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:36.733764 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 32487
I20260927 18:44:36.750537 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 32621
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:44:36.766136 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.751416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.31.26.194:45197)
0927 18:44:36.751591 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.751596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.751614 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:44:36.751711 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.751715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.751783 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.751867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.753080 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.753879 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.755170 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.755569 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.755694 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.760280 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.760296 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.760314 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.760353 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.763349 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.764153 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.764161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.764163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.764300 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.764683 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.764686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.764688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.765054 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.765426 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.765705 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.765884 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:44:36.767079 32744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:36.855422 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 32621
I20260927 18:44:36.871755 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 32754
Sep 27 18:44:36 dist-test-slave-rfd5 krb5kdc[32381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:44:36.887408 427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:36.872598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.31.26.195:46507)
0927 18:44:36.872766 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:36.872771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:36.872810 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:44:36.872983 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:36.872986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:36.873048 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:36.873228 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:36.874664 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.875364 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.876824 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:36.877381 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:36.877479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:36.882136 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:36.882155 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:36.882172 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:36.882213 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:36.884815 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.885616 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.885624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.885627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.885774 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:36.886139 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:36.886144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:36.886146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:36.886499 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:36.886728 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:36.886974 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 18:44:36.887139 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:44:36.888051 409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:36.976857 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 32754
I20260927 18:44:36.995282 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 32394
I20260927 18:44:36.996356 32456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:37.102893 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 32394
2026-09-27T18:44:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4219 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:37 dist-test-slave-rfd5 krb5kdc[462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:37 dist-test-slave-rfd5 krb5kdc[462](info): set up 2 sockets
Sep 27 18:44:37 dist-test-slave-rfd5 krb5kdc[462](info): commencing operation
krb5kdc: starting...
W20260927 18:44:38.213294 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:38 dist-test-slave-rfd5 krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:38.392937 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41753
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:38.717440 475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:38.717746 475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:38.717833 475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:38.725167 475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:38.725291 475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:44:38.725332 475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:38.725389 475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:38.725440 475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:38.725502 475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:38.735692 475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41753
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:38.737025 475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:38.738369 475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:38.750648 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:38.750708 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:38.750681 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:38.751775 475 server_base.cc:1034] running on GCE node
I20260927 18:44:38.752619 475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:38.754184 475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:38.755440 475 hybrid_clock.cc:648] HybridClock initialized: now 1790534678755375 us; error 52 us; skew 500 ppm
I20260927 18:44:38.755795 475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:38 dist-test-slave-rfd5 krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:38.764777 475 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:38.766489 475 webserver.cc:460] Webserver started at http://127.31.26.254:39399/ using document root <none> and password file <none>
I20260927 18:44:38.767222 475 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:38.767313 475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:38.767594 475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:38.770619 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "a26508361dba4c4a8c438f5dab401171"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-rfd5"
server_key: "910d44ecfca0a49749555e9b7a87ff69"
server_key_iv: "034fe0bf241fa9aac5e1e790957ab716"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.771386 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "a26508361dba4c4a8c438f5dab401171"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-rfd5"
server_key: "910d44ecfca0a49749555e9b7a87ff69"
server_key_iv: "034fe0bf241fa9aac5e1e790957ab716"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.776687 475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:44:38.780315 490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:38.781827 475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:38.782023 475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "a26508361dba4c4a8c438f5dab401171"
format_stamp: "Formatted at 2026-09-27 18:44:38 on dist-test-slave-rfd5"
server_key: "910d44ecfca0a49749555e9b7a87ff69"
server_key_iv: "034fe0bf241fa9aac5e1e790957ab716"
server_key_version: "encryptionkey@0"
I20260927 18:44:38.782272 475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:38.799958 475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:38.804167 475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:38.804559 475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:38.826777 475 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41753
I20260927 18:44:38.826798 551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41753 every 8 connection(s)
I20260927 18:44:38.828168 475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:38.830732 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 475
I20260927 18:44:38.830974 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:38.831791 31851 external_mini_cluster.cc:1428] Setting key bb276ec6d68a8ebd637f74b150add543
I20260927 18:44:38.833282 552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:44:38 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:38.847764 552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: Bootstrap starting.
I20260927 18:44:38.851042 552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:38.852073 552 log.cc:824] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:38.855522 552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: No bootstrap required, opened a new log
I20260927 18:44:38.855645 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:38.833900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.31.26.254:41753)
0927 18:44:38.834684 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:38.834696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:38.834739 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:44:38.836316 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:38.836338 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:38.836903 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:38.837342 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:38.839091 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.840457 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.842196 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:38.842908 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:38.843046 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:38.847848 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:38.847856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:38.847873 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:38.847943 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:38.851055 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.851764 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.851771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.851777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.851937 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:38.852366 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:38.852371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:38.852374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:38.853132 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:38.853375 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:38.853901 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 18:44:38.854348 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":380,"threads_started":1}
I20260927 18:44:38.863016 552 raft_consensus.cc:348] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26508361dba4c4a8c438f5dab401171" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41753 } }
I20260927 18:44:38.863372 552 raft_consensus.cc:374] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:38.863447 552 raft_consensus.cc:729] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26508361dba4c4a8c438f5dab401171, State: Initialized, Role: FOLLOWER
I20260927 18:44:38.863953 552 consensus_queue.cc:260] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26508361dba4c4a8c438f5dab401171" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41753 } }
I20260927 18:44:38.864154 552 raft_consensus.cc:388] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:38.864280 552 raft_consensus.cc:482] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:38.864475 552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:38.866793 552 raft_consensus.cc:504] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26508361dba4c4a8c438f5dab401171" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41753 } }
I20260927 18:44:38.867359 552 leader_election.cc:302] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a26508361dba4c4a8c438f5dab401171; no voters:
I20260927 18:44:38.867933 552 leader_election.cc:288] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:38.868077 557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:38.868855 557 raft_consensus.cc:686] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [term 1 LEADER]: Becoming Leader. State: Replica: a26508361dba4c4a8c438f5dab401171, State: Running, Role: LEADER
I20260927 18:44:38.869412 557 consensus_queue.cc:237] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26508361dba4c4a8c438f5dab401171" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41753 } }
I20260927 18:44:38.870121 552 sys_catalog.cc:564] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:38.872311 558 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a26508361dba4c4a8c438f5dab401171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26508361dba4c4a8c438f5dab401171" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41753 } } }
I20260927 18:44:38.872506 559 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a26508361dba4c4a8c438f5dab401171. Latest consensus state: current_term: 1 leader_uuid: "a26508361dba4c4a8c438f5dab401171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26508361dba4c4a8c438f5dab401171" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41753 } } }
I20260927 18:44:38.872795 558 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:38.872797 559 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:38.874379 563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:38.880352 563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:38.889729 563 catalog_manager.cc:1347] Generated new cluster ID: 1d644c4dc03f45f3b6dc3d98d7c615cf
I20260927 18:44:38.889864 563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:38.909897 563 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:38.911857 563 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:38.929627 563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: Generated new TSK 0
I20260927 18:44:38.930814 563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:44:38.975278 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41753
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:39.311028 578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:39.311388 578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:39.311561 578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:39.319870 578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:39.320062 578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:39.320235 578 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:44:39.320393 578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:44:39.330832 578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:39.332259 578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:39.333804 578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:39.347632 583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:39.347790 584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:39.347666 586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:39.348745 578 server_base.cc:1034] running on GCE node
I20260927 18:44:39.349450 578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:39.350409 578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:39.351687 578 hybrid_clock.cc:648] HybridClock initialized: now 1790534679351621 us; error 44 us; skew 500 ppm
I20260927 18:44:39.352005 578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:39 dist-test-slave-rfd5 krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:39.355304 578 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:44:39.356969 578 webserver.cc:460] Webserver started at http://127.31.26.193:45391/ using document root <none> and password file <none>
I20260927 18:44:39.357722 578 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:39.357852 578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:39.358170 578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:39.361027 578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "1a1c7732809a43c3ad4c44329e252eb2"
format_stamp: "Formatted at 2026-09-27 18:44:39 on dist-test-slave-rfd5"
server_key: "903c2b0ab22cbf3ab6896210dbefe534"
server_key_iv: "b2b73fb9389bbefcfa27a96d0270f907"
server_key_version: "encryptionkey@0"
I20260927 18:44:39.361894 578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "1a1c7732809a43c3ad4c44329e252eb2"
format_stamp: "Formatted at 2026-09-27 18:44:39 on dist-test-slave-rfd5"
server_key: "903c2b0ab22cbf3ab6896210dbefe534"
server_key_iv: "b2b73fb9389bbefcfa27a96d0270f907"
server_key_version: "encryptionkey@0"
I20260927 18:44:39.367154 578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:44:39.370883 593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:39.372359 578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:44:39.372552 578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "1a1c7732809a43c3ad4c44329e252eb2"
format_stamp: "Formatted at 2026-09-27 18:44:39 on dist-test-slave-rfd5"
server_key: "903c2b0ab22cbf3ab6896210dbefe534"
server_key_iv: "b2b73fb9389bbefcfa27a96d0270f907"
server_key_version: "encryptionkey@0"
I20260927 18:44:39.372798 578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:39.384613 578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:39.388893 578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:39.389261 578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:39.391903 578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:39.392017 578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:39.392172 578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:39.392252 578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:44:39 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:39.416100 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.392485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:58183 (local address 127.31.26.254:41753)
0927 18:44:39.393190 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:39.393197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:39.394451 (+ 1254us) server_negotiation.cc:407] Connection header received
0927 18:44:39.396895 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:39.396904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:39.397016 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:39.397184 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:39.400231 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.401275 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.403153 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.403872 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.404025 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.408933 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:39.408941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:39.408946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:39.408996 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:39.412288 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.413370 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.413379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.413385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.413563 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.414144 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.414149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.414153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.414448 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:39.414603 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:39.415482 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 18:44:39.415712 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":438,"threads_started":1}
I20260927 18:44:39.417707 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.393327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.31.26.193:58183)
0927 18:44:39.394256 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:39.394304 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:39.396467 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:39.397693 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:39.397714 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:39.398854 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:39.399974 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.400009 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.401515 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.401524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.402986 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.402998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.404160 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.404165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.404357 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.405196 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:39.405234 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:39.408649 (+ 3415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:39.412451 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.412465 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.412494 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.413220 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.413728 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.413733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.413738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.413979 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.414599 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:39.414609 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:39.415071 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 18:44:39.415383 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 18:44:39.415962 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260927 18:44:39.441969 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.424987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.31.26.193:46253)
0927 18:44:39.425844 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:39.425863 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:39.426064 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:39.426514 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:39.426519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:39.426956 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:39.427260 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.427270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.428896 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.428904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.430672 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.430687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.431821 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.431830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.431966 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.432695 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:39.432726 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:39.433856 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:39.438235 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.438246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.438253 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.439040 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.439630 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.439635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.439639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.439815 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.440576 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:39.440582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:39.440774 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 18:44:39.440902 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:44:39.441285 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":481,"threads_started":1}
I20260927 18:44:39.442468 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.424269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:46253 (local address 127.31.26.254:41753)
0927 18:44:39.424454 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:39.424461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:39.426002 (+ 1541us) server_negotiation.cc:407] Connection header received
0927 18:44:39.426213 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:39.426221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:39.426334 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:39.426454 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:39.427414 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.428722 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.430830 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.431587 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.431735 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.434063 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:39.434073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:39.434079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:39.434121 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:39.438037 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.439239 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.439246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.439251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.439443 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.439991 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.439997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.440002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.440422 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:39.441667 (+ 1245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:39.441855 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:44:39.442117 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:44:39.443176 578 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:33359
I20260927 18:44:39.443217 713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:33359 every 8 connection(s)
I20260927 18:44:39.444657 578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:39.445648 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 578
I20260927 18:44:39.445896 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:39.446708 31851 external_mini_cluster.cc:1428] Setting key ba160120980695109ca3483af1c5cf1e
I20260927 18:44:39.447535 714 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41753
I20260927 18:44:39.447902 714 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:39.448630 714 heartbeater.cc:511] Master 127.31.26.254:41753 requested a full tablet report, sending...
I20260927 18:44:39.450879 507 ts_manager.cc:194] Registered new tserver with Master: 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193:33359)
I20260927 18:44:39.453048 507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:58183
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:44:39.488263 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41753
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:39.813796 718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:39.814152 718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:39.814255 718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:39.821718 718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:39.821837 718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:39.821923 718 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:44:39.822005 718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:44:39.835824 718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:39.837512 718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:39.839129 718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:39.855261 726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:39.855337 724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:39.855312 723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:39.856143 718 server_base.cc:1034] running on GCE node
I20260927 18:44:39.856809 718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:39.857846 718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:39.859061 718 hybrid_clock.cc:648] HybridClock initialized: now 1790534679858997 us; error 45 us; skew 500 ppm
I20260927 18:44:39.859380 718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:39 dist-test-slave-rfd5 krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:39.862522 718 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:44:39.863991 718 webserver.cc:460] Webserver started at http://127.31.26.194:37117/ using document root <none> and password file <none>
I20260927 18:44:39.864715 718 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:39.864863 718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:39.865250 718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:39.868100 718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "3d88aebc749c4261880c5f378b2ba667"
format_stamp: "Formatted at 2026-09-27 18:44:39 on dist-test-slave-rfd5"
server_key: "560e914b412e0c1b586add4714f30bd4"
server_key_iv: "382841e701eb666b52dfe01c08161461"
server_key_version: "encryptionkey@0"
I20260927 18:44:39.868912 718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "3d88aebc749c4261880c5f378b2ba667"
format_stamp: "Formatted at 2026-09-27 18:44:39 on dist-test-slave-rfd5"
server_key: "560e914b412e0c1b586add4714f30bd4"
server_key_iv: "382841e701eb666b52dfe01c08161461"
server_key_version: "encryptionkey@0"
I20260927 18:44:39.874279 718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:39.877928 733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:39.879298 718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:39.879487 718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "3d88aebc749c4261880c5f378b2ba667"
format_stamp: "Formatted at 2026-09-27 18:44:39 on dist-test-slave-rfd5"
server_key: "560e914b412e0c1b586add4714f30bd4"
server_key_iv: "382841e701eb666b52dfe01c08161461"
server_key_version: "encryptionkey@0"
I20260927 18:44:39.879726 718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:39.895800 718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:39.899893 718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:39.900182 718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:39.902855 718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:39.902977 718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:39.903141 718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:39.903218 718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:44:39 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:39.929925 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.904265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.31.26.194:41637)
0927 18:44:39.905171 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:39.905215 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:39.907072 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:39.907879 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:39.907896 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:39.908744 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:39.909642 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.909677 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.911411 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.911420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.913508 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.913521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.914824 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.914835 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.915214 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.916190 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:39.916227 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:39.920313 (+ 4086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:39.924313 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.924325 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.924354 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.925011 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.925534 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.925539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.925544 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.925881 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.926746 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:39.926762 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:39.927171 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 18:44:39.927595 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 18:44:39.928162 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":374,"threads_started":1}
I20260927 18:44:39.930804 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.903667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:41637 (local address 127.31.26.254:41753)
0927 18:44:39.903856 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:39.903862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:39.905353 (+ 1491us) server_negotiation.cc:407] Connection header received
0927 18:44:39.907382 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:39.907389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:39.907484 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:39.907603 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:39.909870 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.911212 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.913677 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.914498 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.914660 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.920629 (+ 5969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:39.920639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:39.920647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:39.920704 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:39.924093 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.925229 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.925234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.925238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.925390 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.926098 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.926108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.926128 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.926575 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:39.926754 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:39.930350 (+ 3596us) server_negotiation.cc:299] Negotiation successful
0927 18:44:39.930570 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:44:39.951407 718 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:37163
I20260927 18:44:39.951428 852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:37163 every 8 connection(s)
I20260927 18:44:39.952250 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.937328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.31.26.194:59353)
0927 18:44:39.938219 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:39.938246 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:39.938430 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:39.938795 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:39.938799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:39.939252 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:39.939529 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.939538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.940674 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.940679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.942313 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:39.942329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.943115 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.943122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:39.943296 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.944022 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:39.944040 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:39.945298 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:39.949392 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.949397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.949401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.949838 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.950234 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:39.950240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:39.950243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:39.950387 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:39.951206 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:39.951211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:39.951441 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 18:44:39.951600 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 18:44:39.951916 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":566,"threads_started":1}
I20260927 18:44:39.952883 718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:39.953207 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:39.937693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:59353 (local address 127.31.26.254:41753)
0927 18:44:39.937887 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:39.937893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:39.938346 (+ 453us) server_negotiation.cc:407] Connection header received
0927 18:44:39.938554 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:39.938557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:39.938643 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:39.938755 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:39.939677 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.940523 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.942498 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:39.942914 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:39.943048 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:39.945578 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:39.945604 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:39.945610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:39.945662 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:39.949238 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.949969 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.949975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.949979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.950120 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:39.950565 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:39.950571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:39.950577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:39.951077 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:39.951242 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:39.952693 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0927 18:44:39.952874 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:44:39.955164 853 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41753
I20260927 18:44:39.956228 853 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:39.957172 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 718
I20260927 18:44:39.957218 853 heartbeater.cc:511] Master 127.31.26.254:41753 requested a full tablet report, sending...
I20260927 18:44:39.957387 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:39.957962 31851 external_mini_cluster.cc:1428] Setting key 7c24bb616b0426317240f76d3ed921fe
I20260927 18:44:39.959014 507 ts_manager.cc:194] Registered new tserver with Master: 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163)
I20260927 18:44:39.959960 507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:41637
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:44:39.998570 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41753
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:40.326319 857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:40.326643 857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:40.326740 857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:40.334154 857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:40.334291 857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:40.334376 857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:44:40.334488 857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:44:40.344623 857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:40.345955 857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:40.347206 857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:40.360600 862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:40.360589 863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:40.360585 865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:40.361446 857 server_base.cc:1034] running on GCE node
I20260927 18:44:40.362097 857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:40.363054 857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:40.364323 857 hybrid_clock.cc:648] HybridClock initialized: now 1790534680364256 us; error 44 us; skew 500 ppm
I20260927 18:44:40.364632 857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:40 dist-test-slave-rfd5 krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:40.367746 857 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:44:40.369339 857 webserver.cc:460] Webserver started at http://127.31.26.195:37977/ using document root <none> and password file <none>
I20260927 18:44:40.370052 857 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:40.370173 857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:40.370496 857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:40.373399 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "a6c09161516541cf9f98477bff720522"
format_stamp: "Formatted at 2026-09-27 18:44:40 on dist-test-slave-rfd5"
server_key: "d145254c25b4139026e441754f02c512"
server_key_iv: "200e9892f5f3e13908ca9f0b1e454092"
server_key_version: "encryptionkey@0"
I20260927 18:44:40.374217 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "a6c09161516541cf9f98477bff720522"
format_stamp: "Formatted at 2026-09-27 18:44:40 on dist-test-slave-rfd5"
server_key: "d145254c25b4139026e441754f02c512"
server_key_iv: "200e9892f5f3e13908ca9f0b1e454092"
server_key_version: "encryptionkey@0"
I20260927 18:44:40.379489 857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:44:40.383023 872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:40.384521 857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:44:40.384688 857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "a6c09161516541cf9f98477bff720522"
format_stamp: "Formatted at 2026-09-27 18:44:40 on dist-test-slave-rfd5"
server_key: "d145254c25b4139026e441754f02c512"
server_key_iv: "200e9892f5f3e13908ca9f0b1e454092"
server_key_version: "encryptionkey@0"
I20260927 18:44:40.384899 857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:40.414356 857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:40.419191 857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:40.419489 857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:40.422605 857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:40.422717 857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:40.422860 857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:40.422957 857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:44:40 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:40.448529 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.423154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:47081 (local address 127.31.26.254:41753)
0927 18:44:40.423389 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.423398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.425057 (+ 1659us) server_negotiation.cc:407] Connection header received
0927 18:44:40.427347 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.427352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.427460 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.427577 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:40.430082 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.431169 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.434415 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.434972 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.435123 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.440169 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:40.440180 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:40.440188 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:40.440282 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:40.444164 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.445249 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.445256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.445260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.445416 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.445945 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.445963 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.445966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.446309 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:40.446460 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.447554 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.447867 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 18:44:40.449429 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.423929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.31.26.195:47081)
0927 18:44:40.424862 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.424909 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.427043 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.427851 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.427867 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.428785 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:40.429889 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.429920 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.431335 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.431341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.434223 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.434245 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.435447 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.435457 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.435808 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.436757 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:40.436782 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:40.439873 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:40.444346 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:40.444355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:40.444372 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:40.445053 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:40.445552 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:40.445556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:40.445564 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:40.445809 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:40.446542 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:40.446551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:40.447040 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.447329 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.447847 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":433,"threads_started":1}
I20260927 18:44:40.456552 714 heartbeater.cc:503] Master 127.31.26.254:41753 was elected leader, sending a full tablet report...
I20260927 18:44:40.472163 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.455643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:39269 (local address 127.31.26.254:41753)
0927 18:44:40.455865 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.455874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.456290 (+ 416us) server_negotiation.cc:407] Connection header received
0927 18:44:40.456468 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.456476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.456576 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.456695 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:40.458309 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.459025 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.461350 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.461955 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.462092 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.464793 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:40.464798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:40.464802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:40.464830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:40.468567 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.469677 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.469686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.469690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.469851 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.470347 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.470352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.470356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.470756 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:40.470905 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.471408 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.471696 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:44:40.472361 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.455580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.31.26.195:39269)
0927 18:44:40.456191 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.456211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.456315 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.456902 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.456910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.457702 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:40.458158 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.458174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.459164 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.459172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.461188 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.461206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.462175 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.462184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.462465 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.463197 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:40.463230 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:40.464550 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:40.468747 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:40.468754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:40.468760 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:40.469492 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:40.469970 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:40.469978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:40.469983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:40.470200 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:40.470866 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:40.470871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:40.471070 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.471254 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.471751 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":327,"threads_started":1}
I20260927 18:44:40.482120 857 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:45417
I20260927 18:44:40.482136 991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:45417 every 8 connection(s)
I20260927 18:44:40.483318 857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:40.485466 992 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41753
I20260927 18:44:40.485716 992 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:40.486436 992 heartbeater.cc:511] Master 127.31.26.254:41753 requested a full tablet report, sending...
I20260927 18:44:40.487752 507 ts_manager.cc:194] Registered new tserver with Master: a6c09161516541cf9f98477bff720522 (127.31.26.195:45417)
I20260927 18:44:40.488607 507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:47081
I20260927 18:44:40.490178 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 857
I20260927 18:44:40.490367 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:40.490842 31851 external_mini_cluster.cc:1428] Setting key fb6f0f660f9e39ba0cce6b5f6528ef38
I20260927 18:44:40.493602 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:44:40 dist-test-slave-rfd5 krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:44:40 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:40.538347 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.524586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.31.26.254:41753)
0927 18:44:40.524827 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.524835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.525134 (+ 299us) server_negotiation.cc:407] Connection header received
0927 18:44:40.525356 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.525362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.525435 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.525544 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:40.526960 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.527702 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.528982 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.529524 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.529662 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.533572 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:40.533580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:40.533585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:40.533631 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:40.535956 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.536556 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.536559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.536561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.536683 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:40.537016 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:40.537019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:40.537021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:40.537358 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:40.537539 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.537782 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.537948 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:44:40.542503 507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:40.549481 507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:40.572602 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.562241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.31.26.195:45417)
0927 18:44:40.563661 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.563672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.563709 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:44:40.563805 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.563813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.564060 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.564262 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:40.564939 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.566661 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.569603 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.571082 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.571218 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.571452 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.572071 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.572242 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":1061,"threads_started":1}
I20260927 18:44:40.572654 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.561661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45417 (local address 127.0.0.1:53228)
0927 18:44:40.562725 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.562769 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.562920 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.564269 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.564276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.564306 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:40.564693 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.564708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.566796 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.566801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.569408 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.569418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.571591 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.571597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.571822 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.571880 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.572020 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.572226 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":41,"thread_start_us":747,"threads_started":1}
I20260927 18:44:40.573824 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.564306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37163 (local address 127.0.0.1:41824)
0927 18:44:40.565994 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.566011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.566121 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.567054 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.567058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.567081 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:40.567522 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.567536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.569356 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.569361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.571076 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.571088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.573036 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.573042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.573254 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.573286 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.573359 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.573473 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1584,"mutex_wait_us":25,"thread_start_us":1460,"threads_started":1}
I20260927 18:44:40.573900 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.562215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.31.26.193:33359)
0927 18:44:40.563888 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.563896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.564431 (+ 535us) server_negotiation.cc:407] Connection header received
0927 18:44:40.564533 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.564540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.564780 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.564963 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:40.566009 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.567421 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.569914 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.571443 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.571586 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.571780 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.572181 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.573604 (+ 1423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539,"thread_start_us":1410,"threads_started":1}
I20260927 18:44:40.574245 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.564768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.31.26.194:37163)
0927 18:44:40.566551 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.566559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.566584 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:44:40.566682 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.566690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.566884 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.567077 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:40.567736 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.569187 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.571256 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.572582 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.573369 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.573551 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.573629 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.573778 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":1536,"threads_started":1}
I20260927 18:44:40.574414 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.561675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33359 (local address 127.0.0.1:49740)
0927 18:44:40.564167 (+ 2492us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.564186 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.564302 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.565171 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.565178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.565210 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:40.565699 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.565713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.567547 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.567554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.569752 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.569764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.571663 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.571669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.571877 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.571911 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.572000 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.573241 (+ 1241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2362,"mutex_wait_us":1078,"thread_start_us":2223,"threads_started":1}
I20260927 18:44:40.576862 922 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 (DEFAULT_TABLE table=test-table [id=8a03d045a824496093ac0c7268a1f027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:40.577787 783 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 (DEFAULT_TABLE table=test-table [id=8a03d045a824496093ac0c7268a1f027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:40.578369 922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.578568 644 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 (DEFAULT_TABLE table=test-table [id=8a03d045a824496093ac0c7268a1f027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:40.579161 783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.579838 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.591065 1011 tablet_bootstrap.cc:492] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Bootstrap starting.
I20260927 18:44:40.594300 1012 tablet_bootstrap.cc:492] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Bootstrap starting.
I20260927 18:44:40.596290 1011 tablet_bootstrap.cc:654] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:40.597728 1011 log.cc:824] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:40.597936 1013 tablet_bootstrap.cc:492] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Bootstrap starting.
I20260927 18:44:40.598233 1012 tablet_bootstrap.cc:654] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:40.599867 1012 log.cc:824] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:40.601218 1013 tablet_bootstrap.cc:654] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:40.602494 1013 log.cc:824] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:40.603142 1012 tablet_bootstrap.cc:492] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: No bootstrap required, opened a new log
I20260927 18:44:40.603456 1012 ts_tablet_manager.cc:1397] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 18:44:40.603487 1011 tablet_bootstrap.cc:492] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: No bootstrap required, opened a new log
I20260927 18:44:40.603753 1011 ts_tablet_manager.cc:1397] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260927 18:44:40.605590 1013 tablet_bootstrap.cc:492] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: No bootstrap required, opened a new log
I20260927 18:44:40.605844 1013 ts_tablet_manager.cc:1397] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 18:44:40.611439 1011 raft_consensus.cc:348] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.611820 1011 raft_consensus.cc:374] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.611910 1011 raft_consensus.cc:729] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c09161516541cf9f98477bff720522, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.612519 1011 consensus_queue.cc:260] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.613678 992 heartbeater.cc:503] Master 127.31.26.254:41753 was elected leader, sending a full tablet report...
I20260927 18:44:40.614203 1011 ts_tablet_manager.cc:1428] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 18:44:40.614550 1012 raft_consensus.cc:348] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.614952 1012 raft_consensus.cc:374] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.615031 1012 raft_consensus.cc:729] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1c7732809a43c3ad4c44329e252eb2, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.617350 1012 consensus_queue.cc:260] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.617671 1013 raft_consensus.cc:348] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.618117 1013 raft_consensus.cc:374] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.618237 1013 raft_consensus.cc:729] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d88aebc749c4261880c5f378b2ba667, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.618820 1013 consensus_queue.cc:260] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.622120 853 heartbeater.cc:503] Master 127.31.26.254:41753 was elected leader, sending a full tablet report...
I20260927 18:44:40.622567 1013 ts_tablet_manager.cc:1428] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260927 18:44:40.624477 1012 ts_tablet_manager.cc:1428] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Time spent starting tablet: real 0.021s user 0.015s sys 0.005s
W20260927 18:44:40.698714 715 tablet.cc:2369] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:40.700167 1018 raft_consensus.cc:482] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:40.700498 1018 raft_consensus.cc:504] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.702204 1018 leader_election.cc:288] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6c09161516541cf9f98477bff720522 (127.31.26.195:45417), 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163)
W20260927 18:44:40.705229 854 tablet.cc:2369] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:40.711129 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.702708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:57149 (local address 127.31.26.194:37163)
0927 18:44:40.702959 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.702968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.703877 (+ 909us) server_negotiation.cc:407] Connection header received
0927 18:44:40.704047 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.704052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.704128 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.704235 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:40.704906 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.706043 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.708316 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.709542 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.709726 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.709790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.710728 (+ 938us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.710863 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:44:40.711138 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.702747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:33621 (local address 127.31.26.195:45417)
0927 18:44:40.702970 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.702979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.703699 (+ 720us) server_negotiation.cc:407] Connection header received
0927 18:44:40.703860 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.703865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.703945 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.704043 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:40.704737 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.705654 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.708211 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.709492 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.709676 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.709754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.710728 (+ 974us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.710873 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:44:40.711323 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.702674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37163 (local address 127.31.26.193:57149)
0927 18:44:40.703717 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.703735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.703895 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.704262 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.704265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.704283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:40.704720 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.704731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.706180 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.706185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.708173 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.708184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.710246 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.710251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.710487 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.710584 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.710671 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.710873 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":100,"thread_start_us":506,"threads_started":1}
I20260927 18:44:40.711795 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.702568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45417 (local address 127.31.26.193:33621)
0927 18:44:40.703576 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:40.703593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:40.703726 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:40.704068 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:40.704072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:40.704094 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:40.704574 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.704588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.705787 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.705793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.708035 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:40.708050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.710221 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.710235 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:40.710492 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.710569 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 18:44:40.710671 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:44:40.710873 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":144,"thread_start_us":612,"threads_started":1}
I20260927 18:44:40.712399 803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac96b8a3ca1468e9a4f5d9d0207e891" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d88aebc749c4261880c5f378b2ba667" is_pre_election: true
I20260927 18:44:40.712803 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac96b8a3ca1468e9a4f5d9d0207e891" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c09161516541cf9f98477bff720522" is_pre_election: true
I20260927 18:44:40.712997 803 raft_consensus.cc:2454] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 0.
I20260927 18:44:40.713373 947 raft_consensus.cc:2454] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 0.
I20260927 18:44:40.713821 594 leader_election.cc:302] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1c7732809a43c3ad4c44329e252eb2, 3d88aebc749c4261880c5f378b2ba667; no voters:
I20260927 18:44:40.714229 1018 raft_consensus.cc:2781] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:40.714429 1018 raft_consensus.cc:482] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:40.714550 1018 raft_consensus.cc:3037] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:40.716732 1018 raft_consensus.cc:504] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.717460 1018 leader_election.cc:288] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 election: Requested vote from peers a6c09161516541cf9f98477bff720522 (127.31.26.195:45417), 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163)
I20260927 18:44:40.717986 803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac96b8a3ca1468e9a4f5d9d0207e891" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d88aebc749c4261880c5f378b2ba667"
I20260927 18:44:40.718076 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac96b8a3ca1468e9a4f5d9d0207e891" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c09161516541cf9f98477bff720522"
I20260927 18:44:40.718225 803 raft_consensus.cc:3037] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:40.718266 947 raft_consensus.cc:3037] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:40.721546 803 raft_consensus.cc:2454] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 1.
I20260927 18:44:40.721546 947 raft_consensus.cc:2454] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 1.
I20260927 18:44:40.722119 597 leader_election.cc:302] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a1c7732809a43c3ad4c44329e252eb2, 3d88aebc749c4261880c5f378b2ba667, a6c09161516541cf9f98477bff720522; no voters:
I20260927 18:44:40.722513 1018 raft_consensus.cc:2781] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:40.722972 1018 raft_consensus.cc:686] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a1c7732809a43c3ad4c44329e252eb2, State: Running, Role: LEADER
I20260927 18:44:40.723914 1018 consensus_queue.cc:237] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } }
I20260927 18:44:40.728891 507 catalog_manager.cc:5495] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193). New cstate: current_term: 1 leader_uuid: "1a1c7732809a43c3ad4c44329e252eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } health_report { overall_health: UNKNOWN } } }
W20260927 18:44:40.734978 993 tablet.cc:2369] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:40.774372 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:40.768700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.31.26.193:33359)
0927 18:44:40.768927 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:40.768935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:40.768957 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:44:40.769374 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:40.769378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:40.769454 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:40.769613 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:40.769991 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.770744 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.772153 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:40.772543 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:40.772681 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:40.773205 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:40.773726 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:40.773880 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:40.774015 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:44:40.774094 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:44:40.783321 803 raft_consensus.cc:1261] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Refusing update from remote peer 1a1c7732809a43c3ad4c44329e252eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:40.783321 947 raft_consensus.cc:1261] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Refusing update from remote peer 1a1c7732809a43c3ad4c44329e252eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:40.784557 1018 consensus_queue.cc:1035] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:40.784910 1022 consensus_queue.cc:1035] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:40.796861 1026 mvcc.cc:205] Tried to move back new op lower bound from 7334030052475731968 to 7334030052249120768. Current Snapshot: MvccSnapshot[applied={T|T < 7334030052475731968}]
I20260927 18:44:40.815050 507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:40.815308 507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:40.819103 507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: Sending DeleteTablet for 3 replicas of tablet 3ac96b8a3ca1468e9a4f5d9d0207e891
I20260927 18:44:40.820430 783 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:40 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:41824
I20260927 18:44:40.820433 922 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:40 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:53228
I20260927 18:44:40.821254 644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:40 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:49740
I20260927 18:44:40.822285 507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:40.823109 507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:40.825536 1036 tablet_replica.cc:330] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: stopping tablet replica
I20260927 18:44:40.826037 1036 raft_consensus.cc:2229] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:40.826858 1036 raft_consensus.cc:2258] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:40.833173 782 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 (DEFAULT_TABLE table=test-table [id=94c829e25cd14bfe9d7999bee910d137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:40.833173 921 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 (DEFAULT_TABLE table=test-table [id=94c829e25cd14bfe9d7999bee910d137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:40.833799 782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.833811 921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.840445 1011 tablet_bootstrap.cc:492] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Bootstrap starting.
I20260927 18:44:40.840808 644 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 (DEFAULT_TABLE table=test-table [id=94c829e25cd14bfe9d7999bee910d137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:40.841806 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:40.842567 1011 tablet_bootstrap.cc:654] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:40.851043 1036 ts_tablet_manager.cc:1905] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:40.851603 1013 tablet_bootstrap.cc:492] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Bootstrap starting.
I20260927 18:44:40.853972 1013 tablet_bootstrap.cc:654] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:40.854660 1034 tablet_replica.cc:330] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: stopping tablet replica
I20260927 18:44:40.855680 1034 raft_consensus.cc:2229] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:40.857028 1034 raft_consensus.cc:2258] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:40.860050 1034 ts_tablet_manager.cc:1905] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:40.860634 1036 ts_tablet_manager.cc:1918] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:40.860831 1036 log.cc:1196] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/3ac96b8a3ca1468e9a4f5d9d0207e891
I20260927 18:44:40.861474 1036 ts_tablet_manager.cc:1939] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 1a1c7732809a43c3ad4c44329e252eb2: Deleting consensus metadata
I20260927 18:44:40.863384 491 catalog_manager.cc:4841] TS 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193:33359): tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 (table test-table [id=8a03d045a824496093ac0c7268a1f027]) successfully deleted
I20260927 18:44:40.865446 1035 tablet_replica.cc:330] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: stopping tablet replica
I20260927 18:44:40.865878 1035 raft_consensus.cc:2229] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:40.866375 1035 raft_consensus.cc:2258] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:40.868883 1035 ts_tablet_manager.cc:1905] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:40.872143 1012 tablet_bootstrap.cc:492] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Bootstrap starting.
I20260927 18:44:40.873979 1012 tablet_bootstrap.cc:654] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:40.875253 1034 ts_tablet_manager.cc:1918] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:40.875470 1034 log.cc:1196] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/3ac96b8a3ca1468e9a4f5d9d0207e891
I20260927 18:44:40.876116 1034 ts_tablet_manager.cc:1939] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P 3d88aebc749c4261880c5f378b2ba667: Deleting consensus metadata
I20260927 18:44:40.877008 1035 ts_tablet_manager.cc:1918] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:44:40.877261 1035 log.cc:1196] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/3ac96b8a3ca1468e9a4f5d9d0207e891
I20260927 18:44:40.877861 1035 ts_tablet_manager.cc:1939] T 3ac96b8a3ca1468e9a4f5d9d0207e891 P a6c09161516541cf9f98477bff720522: Deleting consensus metadata
I20260927 18:44:40.878376 491 catalog_manager.cc:4841] TS 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163): tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 (table test-table [id=8a03d045a824496093ac0c7268a1f027]) successfully deleted
I20260927 18:44:40.879552 1011 tablet_bootstrap.cc:492] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: No bootstrap required, opened a new log
I20260927 18:44:40.879733 1011 ts_tablet_manager.cc:1397] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.010s
I20260927 18:44:40.879733 494 catalog_manager.cc:4841] TS a6c09161516541cf9f98477bff720522 (127.31.26.195:45417): tablet 3ac96b8a3ca1468e9a4f5d9d0207e891 (table test-table [id=8a03d045a824496093ac0c7268a1f027]) successfully deleted
I20260927 18:44:40.880800 1011 raft_consensus.cc:348] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:40.881153 1011 raft_consensus.cc:374] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.881273 1011 raft_consensus.cc:729] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c09161516541cf9f98477bff720522, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.881593 1011 consensus_queue.cc:260] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:40.882474 1011 ts_tablet_manager.cc:1428] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:44:40.884096 1013 tablet_bootstrap.cc:492] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: No bootstrap required, opened a new log
I20260927 18:44:40.884232 1013 ts_tablet_manager.cc:1397] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.001s
I20260927 18:44:40.885020 1013 raft_consensus.cc:348] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:40.885267 1013 raft_consensus.cc:374] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.885349 1013 raft_consensus.cc:729] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d88aebc749c4261880c5f378b2ba667, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.885620 1013 consensus_queue.cc:260] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:40.888331 1013 ts_tablet_manager.cc:1428] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 18:44:40.889834 1012 tablet_bootstrap.cc:492] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: No bootstrap required, opened a new log
I20260927 18:44:40.889984 1012 ts_tablet_manager.cc:1397] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.011s
I20260927 18:44:40.890780 1012 raft_consensus.cc:348] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:40.891012 1012 raft_consensus.cc:374] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:40.891115 1012 raft_consensus.cc:729] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1c7732809a43c3ad4c44329e252eb2, State: Initialized, Role: FOLLOWER
I20260927 18:44:40.891359 1012 consensus_queue.cc:260] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:40.892040 1012 ts_tablet_manager.cc:1428] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:44:41.127293 1018 raft_consensus.cc:482] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:41.127519 1018 raft_consensus.cc:504] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.128223 1018 leader_election.cc:288] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163), a6c09161516541cf9f98477bff720522 (127.31.26.195:45417)
I20260927 18:44:41.128780 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec9ec3c0bcd4141a5c6e6b688156e45" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c09161516541cf9f98477bff720522" is_pre_election: true
I20260927 18:44:41.128834 803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec9ec3c0bcd4141a5c6e6b688156e45" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d88aebc749c4261880c5f378b2ba667" is_pre_election: true
I20260927 18:44:41.129030 947 raft_consensus.cc:2454] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 0.
I20260927 18:44:41.129154 803 raft_consensus.cc:2454] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 0.
I20260927 18:44:41.129565 597 leader_election.cc:302] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1c7732809a43c3ad4c44329e252eb2, a6c09161516541cf9f98477bff720522; no voters:
I20260927 18:44:41.129884 1018 raft_consensus.cc:2781] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:41.130020 1018 raft_consensus.cc:482] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:41.130139 1018 raft_consensus.cc:3037] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:41.132513 1018 raft_consensus.cc:504] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.133363 1018 leader_election.cc:288] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163), a6c09161516541cf9f98477bff720522 (127.31.26.195:45417)
I20260927 18:44:41.133767 803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec9ec3c0bcd4141a5c6e6b688156e45" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d88aebc749c4261880c5f378b2ba667"
I20260927 18:44:41.134023 803 raft_consensus.cc:3037] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:41.133949 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec9ec3c0bcd4141a5c6e6b688156e45" candidate_uuid: "1a1c7732809a43c3ad4c44329e252eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c09161516541cf9f98477bff720522"
I20260927 18:44:41.134193 947 raft_consensus.cc:3037] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:41.136086 803 raft_consensus.cc:2454] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 1.
I20260927 18:44:41.136552 594 leader_election.cc:302] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1c7732809a43c3ad4c44329e252eb2, 3d88aebc749c4261880c5f378b2ba667; no voters:
I20260927 18:44:41.136883 1018 raft_consensus.cc:2781] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:41.137063 1018 raft_consensus.cc:686] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a1c7732809a43c3ad4c44329e252eb2, State: Running, Role: LEADER
I20260927 18:44:41.137198 947 raft_consensus.cc:2454] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a1c7732809a43c3ad4c44329e252eb2 in term 1.
I20260927 18:44:41.137372 1018 consensus_queue.cc:237] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.140336 506 catalog_manager.cc:5495] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193). New cstate: current_term: 1 leader_uuid: "1a1c7732809a43c3ad4c44329e252eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:44:41 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534678, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:41.247031 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.231880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.31.26.254:41753)
0927 18:44:41.233431 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.233438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.233462 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:44:41.233517 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.233520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.233574 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.233687 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:41.234880 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.235675 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.237067 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.237528 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.237672 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.241636 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:41.241641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:41.241645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:41.241673 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:41.244240 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.244940 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.244948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.244952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.245070 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.245498 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.245505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.245508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.245774 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:41.245924 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.246311 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.246584 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":1437,"thread_start_us":1303,"threads_started":1}
I20260927 18:44:41.247097 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.231867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41753 (local address 127.0.0.1:35814)
0927 18:44:41.233167 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:41.233193 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:41.233373 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:41.233718 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:41.233748 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:41.234451 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:41.234702 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.234712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.235801 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.235809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.236926 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.236939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.237696 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.237702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.237877 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.238592 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:41.238630 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:41.241435 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:41.244394 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:41.244406 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:41.244415 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:41.244821 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:41.245238 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:41.245244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:41.245248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:41.245397 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:41.245899 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:41.245908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:41.246069 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 18:44:41.246230 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:44:41.246711 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":162,"thread_start_us":908,"threads_started":1}
I20260927 18:44:41.252921 506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:35814:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:44:41.254048 506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:41.261447 644 tablet_service.cc:1467] Processing CreateTablet for tablet 8348078d77914bbfbbb83f54c00e47cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8df5d72dee5d460495d5950b720aebb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:44:41.261761 782 tablet_service.cc:1467] Processing CreateTablet for tablet 8348078d77914bbfbbb83f54c00e47cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8df5d72dee5d460495d5950b720aebb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:44:41.262091 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8348078d77914bbfbbb83f54c00e47cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:41.262346 782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8348078d77914bbfbbb83f54c00e47cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:41.262256 922 tablet_service.cc:1467] Processing CreateTablet for tablet 8348078d77914bbfbbb83f54c00e47cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8df5d72dee5d460495d5950b720aebb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:44:41.262754 922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8348078d77914bbfbbb83f54c00e47cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:41.270004 1011 tablet_bootstrap.cc:492] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: Bootstrap starting.
I20260927 18:44:41.270658 1012 tablet_bootstrap.cc:492] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: Bootstrap starting.
I20260927 18:44:41.271091 1013 tablet_bootstrap.cc:492] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: Bootstrap starting.
I20260927 18:44:41.271845 1011 tablet_bootstrap.cc:654] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:41.272512 1012 tablet_bootstrap.cc:654] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:41.272631 1013 tablet_bootstrap.cc:654] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:41.281627 1012 tablet_bootstrap.cc:492] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: No bootstrap required, opened a new log
I20260927 18:44:41.281855 1012 ts_tablet_manager.cc:1397] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.002s
I20260927 18:44:41.282096 1013 tablet_bootstrap.cc:492] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: No bootstrap required, opened a new log
I20260927 18:44:41.282311 1013 ts_tablet_manager.cc:1397] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260927 18:44:41.282744 1011 tablet_bootstrap.cc:492] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: No bootstrap required, opened a new log
I20260927 18:44:41.282723 1012 raft_consensus.cc:348] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.282917 1011 ts_tablet_manager.cc:1397] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260927 18:44:41.282997 1012 raft_consensus.cc:374] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:41.283085 1012 raft_consensus.cc:729] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1c7732809a43c3ad4c44329e252eb2, State: Initialized, Role: FOLLOWER
I20260927 18:44:41.283085 1013 raft_consensus.cc:348] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.283354 1013 raft_consensus.cc:374] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:41.283455 1013 raft_consensus.cc:729] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d88aebc749c4261880c5f378b2ba667, State: Initialized, Role: FOLLOWER
I20260927 18:44:41.283331 1012 consensus_queue.cc:260] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.283787 1011 raft_consensus.cc:348] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.283761 1018 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.284019 1011 raft_consensus.cc:374] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:41.284093 1018 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.284087 1011 raft_consensus.cc:729] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c09161516541cf9f98477bff720522, State: Initialized, Role: FOLLOWER
I20260927 18:44:41.283751 1013 consensus_queue.cc:260] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.284372 1011 consensus_queue.cc:260] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.284796 1019 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.284941 1017 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.285283 1017 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.285307 1012 ts_tablet_manager.cc:1428] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:44:41.285713 1011 ts_tablet_manager.cc:1428] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 18:44:41.285729 1019 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.287907 1013 ts_tablet_manager.cc:1428] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260927 18:44:41.475440 1019 raft_consensus.cc:482] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:41.475723 1019 raft_consensus.cc:504] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.477362 1019 leader_election.cc:288] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193:33359), a6c09161516541cf9f98477bff720522 (127.31.26.195:45417)
I20260927 18:44:41.497044 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.477878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:60175 (local address 127.31.26.193:33359)
0927 18:44:41.490260 (+ 12382us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.490268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.490837 (+ 569us) server_negotiation.cc:407] Connection header received
0927 18:44:41.490916 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.490923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.491000 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.491088 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:41.491933 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.492989 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.495249 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.496005 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.496205 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.496283 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.496631 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.496765 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12229,"thread_start_us":12082,"threads_started":1}
I20260927 18:44:41.497188 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.477532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33359 (local address 127.31.26.194:60175)
0927 18:44:41.490260 (+ 12728us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:41.490286 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:41.490489 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:41.491207 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:41.491215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:41.491253 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:41.491701 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.491715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.493160 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.493165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.495102 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.495113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.496197 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.496202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.496418 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.496461 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:44:41.496556 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:44:41.496718 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12594,"mutex_wait_us":34,"thread_start_us":13166,"threads_started":1}
I20260927 18:44:41.497335 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.477773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45417 (local address 127.31.26.194:58035)
0927 18:44:41.490441 (+ 12668us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:41.490461 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:41.490602 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:41.490992 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:41.490997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:41.491024 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:41.491326 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.491341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.492690 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.492697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.494759 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.494774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.496464 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.496469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.496670 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.496699 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:44:41.496773 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:44:41.496918 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12538,"mutex_wait_us":26,"thread_start_us":12822,"threads_started":1}
I20260927 18:44:41.498026 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.477788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:58035 (local address 127.31.26.195:45417)
0927 18:44:41.490658 (+ 12870us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.490666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.490692 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:44:41.490761 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.490765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.490846 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.490952 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:41.491523 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.492539 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.494950 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.496275 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.496458 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.496529 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.497570 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.497713 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12705,"thread_start_us":13058,"threads_started":1}
I20260927 18:44:41.498378 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8348078d77914bbfbbb83f54c00e47cc" candidate_uuid: "3d88aebc749c4261880c5f378b2ba667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1c7732809a43c3ad4c44329e252eb2" is_pre_election: true
I20260927 18:44:41.498648 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8348078d77914bbfbbb83f54c00e47cc" candidate_uuid: "3d88aebc749c4261880c5f378b2ba667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c09161516541cf9f98477bff720522" is_pre_election: true
I20260927 18:44:41.498775 669 raft_consensus.cc:2454] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d88aebc749c4261880c5f378b2ba667 in term 0.
I20260927 18:44:41.498852 947 raft_consensus.cc:2454] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d88aebc749c4261880c5f378b2ba667 in term 0.
I20260927 18:44:41.499354 737 leader_election.cc:302] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a1c7732809a43c3ad4c44329e252eb2, 3d88aebc749c4261880c5f378b2ba667, a6c09161516541cf9f98477bff720522; no voters:
I20260927 18:44:41.499786 1019 raft_consensus.cc:2781] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:41.499922 1019 raft_consensus.cc:482] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:41.500016 1019 raft_consensus.cc:3037] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:41.502514 1019 raft_consensus.cc:504] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.503131 1019 leader_election.cc:288] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [CANDIDATE]: Term 1 election: Requested vote from peers 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193:33359), a6c09161516541cf9f98477bff720522 (127.31.26.195:45417)
I20260927 18:44:41.503600 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8348078d77914bbfbbb83f54c00e47cc" candidate_uuid: "3d88aebc749c4261880c5f378b2ba667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1c7732809a43c3ad4c44329e252eb2"
I20260927 18:44:41.503829 669 raft_consensus.cc:3037] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:41.503784 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8348078d77914bbfbbb83f54c00e47cc" candidate_uuid: "3d88aebc749c4261880c5f378b2ba667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c09161516541cf9f98477bff720522"
I20260927 18:44:41.504052 947 raft_consensus.cc:3037] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:41.506217 947 raft_consensus.cc:2454] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d88aebc749c4261880c5f378b2ba667 in term 1.
I20260927 18:44:41.506707 737 leader_election.cc:302] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d88aebc749c4261880c5f378b2ba667, a6c09161516541cf9f98477bff720522; no voters:
I20260927 18:44:41.506867 669 raft_consensus.cc:2454] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d88aebc749c4261880c5f378b2ba667 in term 1.
I20260927 18:44:41.507061 1019 raft_consensus.cc:2781] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:41.507550 1019 raft_consensus.cc:686] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [term 1 LEADER]: Becoming Leader. State: Replica: 3d88aebc749c4261880c5f378b2ba667, State: Running, Role: LEADER
I20260927 18:44:41.508008 1019 consensus_queue.cc:237] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } }
I20260927 18:44:41.510164 1055 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: TxnStatusTablet state changed. Reason: New leader 3d88aebc749c4261880c5f378b2ba667. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.510740 1055 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:44:41.511466 1057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:44:41.511703 1057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:44:41.512148 505 catalog_manager.cc:5495] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20260927 18:44:41.531925 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.525288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.31.26.194:37163)
0927 18:44:41.526075 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.526080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.526096 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:41.526174 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.526178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.526250 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.526375 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:41.526815 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.527636 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.529572 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.530018 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.530124 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.530707 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:41.531307 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:41.531414 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.531580 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.531645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":523,"threads_started":1}
I20260927 18:44:41.532032 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.525261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37163 (local address 127.0.0.1:41830)
0927 18:44:41.525482 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:41.525504 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:41.525630 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:41.526439 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:41.526444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:41.526459 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:41.526652 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.526663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.527770 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.527785 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.529434 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.529449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.530171 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.530178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.530415 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.530497 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:41.531423 (+ 926us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:41.531446 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:44:41.531539 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:44:41.531703 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:44:41.536617 947 raft_consensus.cc:1261] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Refusing update from remote peer 3d88aebc749c4261880c5f378b2ba667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:41.536886 669 raft_consensus.cc:1261] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Refusing update from remote peer 3d88aebc749c4261880c5f378b2ba667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:41.537873 1055 consensus_queue.cc:1035] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 18:44:41.538546 1019 consensus_queue.cc:1035] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:41.543530 1017 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: TxnStatusTablet state changed. Reason: New leader 3d88aebc749c4261880c5f378b2ba667. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.543828 1017 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.546401 1018 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: TxnStatusTablet state changed. Reason: New leader 3d88aebc749c4261880c5f378b2ba667. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.546677 1018 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.550922 1064 mvcc.cc:205] Tried to move back new op lower bound from 7334030055567024128 to 7334030055460663296. Current Snapshot: MvccSnapshot[applied={T|T < 7334030055567024128}]
I20260927 18:44:41.553970 1017 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.554306 1017 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P a6c09161516541cf9f98477bff720522: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.554728 1061 mvcc.cc:205] Tried to move back new op lower bound from 7334030055567024128 to 7334030055460663296. Current Snapshot: MvccSnapshot[applied={T|T < 7334030055567024128}]
I20260927 18:44:41.558228 1018 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.558502 1018 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 1a1c7732809a43c3ad4c44329e252eb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:41.559793 1055 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.560062 1055 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:44:41.560515 1055 tablet_replica.cc:439] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d88aebc749c4261880c5f378b2ba667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1c7732809a43c3ad4c44329e252eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33359 } } peers { permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 } } }
I20260927 18:44:41.560770 1055 tablet_replica.cc:442] T 8348078d77914bbfbbb83f54c00e47cc P 3d88aebc749c4261880c5f378b2ba667: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:44:41.581512 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.575048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:50209 (local address 127.31.26.194:37163)
0927 18:44:41.575232 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.575238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.575817 (+ 579us) server_negotiation.cc:407] Connection header received
0927 18:44:41.576031 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.576059 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.576192 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.576276 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:41.576831 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.577824 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.579679 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.580149 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.580294 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.580727 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:41.580833 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:41.580941 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.581116 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.581196 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:44:41.581612 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.575029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37163 (local address 127.31.26.193:50209)
0927 18:44:41.575710 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:41.575726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:41.575893 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:41.576304 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:41.576309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:41.576323 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:41.576687 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.576698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.577966 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.577974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.579513 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.579526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.580317 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.580323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.580473 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.580533 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:41.580941 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:41.580955 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:44:41.581059 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:44:41.581228 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":346,"threads_started":1}
I20260927 18:44:41.591351 803 raft_consensus.cc:1261] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Refusing update from remote peer 1a1c7732809a43c3ad4c44329e252eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:41.591380 947 raft_consensus.cc:1261] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Refusing update from remote peer 1a1c7732809a43c3ad4c44329e252eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:41.591923 1022 consensus_queue.cc:1035] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d88aebc749c4261880c5f378b2ba667" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:41.592217 1018 consensus_queue.cc:1035] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c09161516541cf9f98477bff720522" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:41.631093 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.624769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:42235 (local address 127.31.26.193:33359)
0927 18:44:41.624942 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.624948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.625689 (+ 741us) server_negotiation.cc:407] Connection header received
0927 18:44:41.625898 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.625903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.625986 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.626081 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:41.626884 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.627573 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.629055 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.629630 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.629755 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.630323 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:41.630464 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:41.630554 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.630748 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.630826 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:44:41.631506 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.624863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33359 (local address 127.31.26.194:42235)
0927 18:44:41.625585 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:41.625602 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:41.625769 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:41.626248 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:41.626253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:41.626265 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:41.626622 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.626631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.627695 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.627703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.628907 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:41.628919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.629834 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.629841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:41.630048 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.630134 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:41.630573 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:41.630589 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:44:41.630719 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 18:44:41.630998 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":1161,"threads_started":1}
I20260927 18:44:41.665843 505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:41.666196 505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:41.669028 505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a26508361dba4c4a8c438f5dab401171: Sending DeleteTablet for 3 replicas of tablet 8ec9ec3c0bcd4141a5c6e6b688156e45
I20260927 18:44:41.669935 782 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:41 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:41824
I20260927 18:44:41.669929 922 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:41 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:53228
I20260927 18:44:41.670753 644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:41 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:49740
I20260927 18:44:41.679150 1081 tablet_replica.cc:330] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: stopping tablet replica
I20260927 18:44:41.679441 1081 raft_consensus.cc:2229] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:41.679955 1081 raft_consensus.cc:2258] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:41.682886 1081 ts_tablet_manager.cc:1905] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:41.685607 1080 tablet_replica.cc:330] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: stopping tablet replica
I20260927 18:44:41.685933 1080 raft_consensus.cc:2229] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:41.686210 1080 raft_consensus.cc:2258] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:41.687330 1080 ts_tablet_manager.cc:1905] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:41.693567 1079 tablet_replica.cc:330] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: stopping tablet replica
I20260927 18:44:41.693771 1079 raft_consensus.cc:2229] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:41.694315 1079 raft_consensus.cc:2258] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:41.695024 1081 ts_tablet_manager.cc:1918] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:41.695159 1081 log.cc:1196] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/8ec9ec3c0bcd4141a5c6e6b688156e45
I20260927 18:44:41.695644 1081 ts_tablet_manager.cc:1939] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 1a1c7732809a43c3ad4c44329e252eb2: Deleting consensus metadata
I20260927 18:44:41.696718 491 catalog_manager.cc:4841] TS 1a1c7732809a43c3ad4c44329e252eb2 (127.31.26.193:33359): tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 (table test-table [id=94c829e25cd14bfe9d7999bee910d137]) successfully deleted
I20260927 18:44:41.697827 1079 ts_tablet_manager.cc:1905] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:41.698252 1080 ts_tablet_manager.cc:1918] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:41.698395 1080 log.cc:1196] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/8ec9ec3c0bcd4141a5c6e6b688156e45
I20260927 18:44:41.698904 1080 ts_tablet_manager.cc:1939] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P a6c09161516541cf9f98477bff720522: Deleting consensus metadata
I20260927 18:44:41.699995 494 catalog_manager.cc:4841] TS a6c09161516541cf9f98477bff720522 (127.31.26.195:45417): tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 (table test-table [id=94c829e25cd14bfe9d7999bee910d137]) successfully deleted
Sep 27 18:44:41 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:41.705307 1079 ts_tablet_manager.cc:1918] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:41.705451 1079 log.cc:1196] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/8ec9ec3c0bcd4141a5c6e6b688156e45
I20260927 18:44:41.705924 1079 ts_tablet_manager.cc:1939] T 8ec9ec3c0bcd4141a5c6e6b688156e45 P 3d88aebc749c4261880c5f378b2ba667: Deleting consensus metadata
I20260927 18:44:41.706985 491 catalog_manager.cc:4841] TS 3d88aebc749c4261880c5f378b2ba667 (127.31.26.194:37163): tablet 8ec9ec3c0bcd4141a5c6e6b688156e45 (table test-table [id=94c829e25cd14bfe9d7999bee910d137]) successfully deleted
I20260927 18:44:41.709612 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.689166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.31.26.193:33359)
0927 18:44:41.689340 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.689346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.690031 (+ 685us) server_negotiation.cc:407] Connection header received
0927 18:44:41.690294 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.690299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.690389 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.690465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:41.692835 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.694037 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.697116 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.697754 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.697900 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.703473 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:41.703486 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:41.703500 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:41.703534 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:41.706486 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.707375 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.707385 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.707389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.707553 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.707980 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.707985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.707989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.708347 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:41.708658 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.708954 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.709197 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:44:41.710417 709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49760
I20260927 18:44:41.727641 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.713348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.31.26.254:41753)
0927 18:44:41.713587 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.713593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.713793 (+ 200us) server_negotiation.cc:407] Connection header received
0927 18:44:41.713936 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.713942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.714019 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.714147 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:41.715322 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.716890 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.718193 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.718834 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.718960 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.721636 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:41.721644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:41.721649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:41.721690 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:41.725014 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.725683 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.725689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.725693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.725832 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.726175 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.726180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.726183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.726536 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:41.726779 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.727045 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.727281 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260927 18:44:41.728266 505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35816
I20260927 18:44:41.730324 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 578
I20260927 18:44:41.744366 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.731446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.31.26.193:33359)
0927 18:44:41.731657 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.731663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.731824 (+ 161us) server_negotiation.cc:407] Connection header received
0927 18:44:41.731938 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.731944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.732016 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.732143 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:41.733322 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.734481 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.735618 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.736265 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.736390 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.738588 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:41.738594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:41.738600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:41.738643 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:41.741896 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.742488 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.742494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.742497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.742641 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.742964 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.742969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.742972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.743303 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:41.743543 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.743798 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.744031 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 18:44:41.744988 709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49764
W20260927 18:44:41.745595 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:41.745798 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 578
I20260927 18:44:41.776093 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 718
Sep 27 18:44:41 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:44:41.793298 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.776912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.31.26.194:37163)
0927 18:44:41.777121 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.777129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.777151 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:44:41.777360 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.777364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.777439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.777530 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:41.779040 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.779773 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.781793 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.782205 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.782350 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.786864 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:41.786877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:41.786894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:41.786934 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:41.790442 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.791134 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.791141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.791143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.791265 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.791653 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.791657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.791659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.792021 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:41.792344 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.792648 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.792892 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:44:41.794159 848 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41832
W20260927 18:44:41.794792 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:41.794981 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 718
I20260927 18:44:41.811105 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 857
Sep 27 18:44:41 dist-test-slave-rfd5 krb5kdc[462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:44:41.826411 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:41.811945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.31.26.195:45417)
0927 18:44:41.812154 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:41.812159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:41.812176 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:44:41.812362 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:41.812366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:41.812447 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:41.812543 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:41.813908 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.815012 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.816326 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:41.816715 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:41.816817 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:41.821005 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:41.821020 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:41.821036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:41.821073 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:41.823963 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.824583 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.824590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.824593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.824715 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:41.825011 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:41.825014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:41.825016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:41.825427 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:41.825641 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:41.825877 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:44:41.826062 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 18:44:41.827209 987 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53240
W20260927 18:44:41.827848 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 857: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:41.828003 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 857
I20260927 18:44:41.851402 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 475
I20260927 18:44:41.852623 547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:41.966547 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 475
2026-09-27T18:44:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4865 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:42 dist-test-slave-rfd5 krb5kdc[1090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:42 dist-test-slave-rfd5 krb5kdc[1090](info): set up 2 sockets
Sep 27 18:44:42 dist-test-slave-rfd5 krb5kdc[1090](info): commencing operation
krb5kdc: starting...
W20260927 18:44:43.061285 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:43 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:43.247956 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41365
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:43.572166 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:43.572563 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:43.572737 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:43.580116 1103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:43.580308 1103 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:44:43.580394 1103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:43.580507 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:43.580607 1103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:43.580698 1103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:43.590919 1103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41365
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:43.592402 1103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:43.593928 1103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:43.606629 1111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:43.606666 1109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:43.607364 1108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:43.607999 1103 server_base.cc:1034] running on GCE node
I20260927 18:44:43.608935 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:43.610471 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:43.611713 1103 hybrid_clock.cc:648] HybridClock initialized: now 1790534683611652 us; error 37 us; skew 500 ppm
I20260927 18:44:43.612097 1103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:43 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:43.615530 1103 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:43.617163 1103 webserver.cc:460] Webserver started at http://127.31.26.254:45911/ using document root <none> and password file <none>
I20260927 18:44:43.617910 1103 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:43.618024 1103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:43.618369 1103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:43.621528 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "9fca1ea600e24b048067b8dac3f30ec3"
format_stamp: "Formatted at 2026-09-27 18:44:43 on dist-test-slave-rfd5"
server_key: "c0a71599b5d053c84edf662bca0d55c8"
server_key_iv: "75e318fbc031d79f46026056eed62b58"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.622339 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "9fca1ea600e24b048067b8dac3f30ec3"
format_stamp: "Formatted at 2026-09-27 18:44:43 on dist-test-slave-rfd5"
server_key: "c0a71599b5d053c84edf662bca0d55c8"
server_key_iv: "75e318fbc031d79f46026056eed62b58"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.628086 1103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:44:43.631752 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:43.633224 1103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:43.633404 1103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "9fca1ea600e24b048067b8dac3f30ec3"
format_stamp: "Formatted at 2026-09-27 18:44:43 on dist-test-slave-rfd5"
server_key: "c0a71599b5d053c84edf662bca0d55c8"
server_key_iv: "75e318fbc031d79f46026056eed62b58"
server_key_version: "encryptionkey@0"
I20260927 18:44:43.633636 1103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:43.652262 1103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:43.656486 1103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:43.656788 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:43.678838 1103 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41365
I20260927 18:44:43.678841 1179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41365 every 8 connection(s)
I20260927 18:44:43.680274 1103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:43.684136 1180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:43.686544 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 1103
I20260927 18:44:43.686800 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:43.687644 31851 external_mini_cluster.cc:1428] Setting key ea8d3fb39ffa79e264f54c01e0277fe2
I20260927 18:44:43.697841 1180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Bootstrap starting.
I20260927 18:44:43.702116 1180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:43.703611 1180 log.cc:824] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:44:43 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:43.706856 1180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: No bootstrap required, opened a new log
I20260927 18:44:43.712723 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:43.690028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.31.26.254:41365)
0927 18:44:43.690952 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:43.690967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:43.691021 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:44:43.692805 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:43.692832 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:43.693386 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:43.693783 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:43.695721 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.697180 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.699015 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:43.699795 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:43.699986 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:43.705195 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:43.705206 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:43.705224 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:43.705262 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:43.708354 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.709129 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.709145 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.709152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.709336 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:43.709793 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:43.709800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:43.709804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:43.710385 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:43.710736 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:43.711212 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 18:44:43.711652 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":454,"threads_started":1}
I20260927 18:44:43.714913 1180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca1ea600e24b048067b8dac3f30ec3" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41365 } }
I20260927 18:44:43.715306 1180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:43.715412 1180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fca1ea600e24b048067b8dac3f30ec3, State: Initialized, Role: FOLLOWER
I20260927 18:44:43.716022 1180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca1ea600e24b048067b8dac3f30ec3" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41365 } }
I20260927 18:44:43.716228 1180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:43.716324 1180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:43.716442 1180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:43.718652 1180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca1ea600e24b048067b8dac3f30ec3" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41365 } }
I20260927 18:44:43.719165 1180 leader_election.cc:302] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fca1ea600e24b048067b8dac3f30ec3; no voters:
I20260927 18:44:43.719936 1180 leader_election.cc:288] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:43.720046 1185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:43.720829 1185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 9fca1ea600e24b048067b8dac3f30ec3, State: Running, Role: LEADER
I20260927 18:44:43.721410 1185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca1ea600e24b048067b8dac3f30ec3" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41365 } }
I20260927 18:44:43.722343 1180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:43.724570 1187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fca1ea600e24b048067b8dac3f30ec3. Latest consensus state: current_term: 1 leader_uuid: "9fca1ea600e24b048067b8dac3f30ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca1ea600e24b048067b8dac3f30ec3" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41365 } } }
I20260927 18:44:43.724413 1186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fca1ea600e24b048067b8dac3f30ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca1ea600e24b048067b8dac3f30ec3" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41365 } } }
I20260927 18:44:43.725392 1187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:43.725504 1186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:43.727437 1194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:43.732308 1194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:43.742368 1194 catalog_manager.cc:1347] Generated new cluster ID: 42487e18202a4b4daf6d942cd235f3d6
I20260927 18:44:43.742503 1194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:43.754187 1194 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:43.756204 1194 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:43.767184 1194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Generated new TSK 0
I20260927 18:44:43.768093 1194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:44:43.816991 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41365
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:44.168411 1206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:44.168704 1206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:44.168883 1206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:44.176342 1206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:44.176460 1206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:44.176664 1206 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:44:44.176750 1206 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:44:44.176887 1206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:44:44.186977 1206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:41365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:44.188349 1206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:44.189858 1206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:44.203339 1211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.203377 1212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.203490 1214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.204131 1206 server_base.cc:1034] running on GCE node
I20260927 18:44:44.205011 1206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:44.206001 1206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:44.207242 1206 hybrid_clock.cc:648] HybridClock initialized: now 1790534684207172 us; error 64 us; skew 500 ppm
I20260927 18:44:44.207582 1206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:44 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:44.216082 1206 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:44:44.217717 1206 webserver.cc:460] Webserver started at http://127.31.26.193:35209/ using document root <none> and password file <none>
I20260927 18:44:44.218453 1206 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:44.218600 1206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:44.218922 1206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:44.221978 1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "905cda65bc8042dbb8967f5032022eaf"
format_stamp: "Formatted at 2026-09-27 18:44:44 on dist-test-slave-rfd5"
server_key: "77ebc5b77f192c01f191cdea2a129107"
server_key_iv: "092ae0b9821837873a85e1126cc85d25"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.222783 1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "905cda65bc8042dbb8967f5032022eaf"
format_stamp: "Formatted at 2026-09-27 18:44:44 on dist-test-slave-rfd5"
server_key: "77ebc5b77f192c01f191cdea2a129107"
server_key_iv: "092ae0b9821837873a85e1126cc85d25"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.228633 1206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:44:44.232085 1221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.233565 1206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:44:44.233735 1206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "905cda65bc8042dbb8967f5032022eaf"
format_stamp: "Formatted at 2026-09-27 18:44:44 on dist-test-slave-rfd5"
server_key: "77ebc5b77f192c01f191cdea2a129107"
server_key_iv: "092ae0b9821837873a85e1126cc85d25"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.233966 1206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:44.261505 1206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:44.266223 1206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:44.266528 1206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:44.269431 1206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:44.269637 1206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.269806 1206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:44.269977 1206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:44:44 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:44.292701 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.269771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:51261 (local address 127.31.26.254:41365)
0927 18:44:44.270239 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:44.270245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:44.271497 (+ 1252us) server_negotiation.cc:407] Connection header received
0927 18:44:44.273691 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:44.273701 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:44.273818 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:44.273937 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:44.276215 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.277339 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.279964 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.280376 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.280516 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.286144 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:44.286153 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:44.286160 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:44.286205 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:44.289588 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.290342 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.290349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.290352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.290473 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.290937 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.290943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.290947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.291212 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:44.291364 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:44.292105 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 18:44:44.292322 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260927 18:44:44.294184 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.270443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.31.26.193:51261)
0927 18:44:44.271313 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.271368 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:44.273378 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:44.274227 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:44.274241 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:44.275043 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:44.276002 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.276033 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.277490 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.277495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.279804 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.279819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.280639 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.280646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.281000 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.281947 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:44.281983 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:44.285797 (+ 3814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:44.289747 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.289758 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.289776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.290214 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.290595 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.290601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.290604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.290820 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.291340 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:44.291348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:44.291728 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 18:44:44.292011 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 18:44:44.292567 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":372,"threads_started":1}
I20260927 18:44:44.321161 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.303874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.31.26.193:38283)
0927 18:44:44.304877 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.304902 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:44.305128 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:44.305688 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:44.305691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:44.306419 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:44.306976 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.306995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.308176 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.308184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.309904 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.309918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.311057 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.311066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.311338 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.312147 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:44.312178 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:44.313407 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:44.318021 (+ 4614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.318028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.318033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.318626 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.319109 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.319116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.319121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.319271 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.320036 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:44.320043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:44.320251 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 18:44:44.320414 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:44:44.320767 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":646,"threads_started":1}
I20260927 18:44:44.322932 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.303902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:38283 (local address 127.31.26.254:41365)
0927 18:44:44.304153 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:44.304163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:44.305008 (+ 845us) server_negotiation.cc:407] Connection header received
0927 18:44:44.305287 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:44.305295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:44.305391 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:44.305497 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:44.307161 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.308037 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.310062 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.310817 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.310961 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.313682 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:44.313690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:44.313698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:44.313736 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:44.317835 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.318793 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.318802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.318806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.318965 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.319410 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.319415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.319419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.319814 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:44.319968 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:44.321996 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0927 18:44:44.322351 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:44:44.326131 1206 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:45917
I20260927 18:44:44.326225 1341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:45917 every 8 connection(s)
I20260927 18:44:44.327419 1206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:44.329440 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 1206
I20260927 18:44:44.329519 1342 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41365
I20260927 18:44:44.329629 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:44.329748 1342 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:44.330296 31851 external_mini_cluster.cc:1428] Setting key 5dc1ef9d5533062bdbbbe7c00038bb2d
I20260927 18:44:44.330436 1342 heartbeater.cc:511] Master 127.31.26.254:41365 requested a full tablet report, sending...
I20260927 18:44:44.332842 1134 ts_manager.cc:194] Registered new tserver with Master: 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917)
I20260927 18:44:44.335161 1134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:51261
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:44:44.372622 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41365
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:44.701144 1346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:44.701444 1346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:44.701548 1346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:44.708595 1346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:44.708702 1346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:44.708787 1346 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:44:44.708841 1346 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:44:44.708902 1346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:44:44.718693 1346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:41365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:44.719965 1346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:44.721268 1346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:44.735627 1351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.735651 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:44.735826 1354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:44.736408 1346 server_base.cc:1034] running on GCE node
I20260927 18:44:44.737149 1346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:44.738332 1346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:44.739583 1346 hybrid_clock.cc:648] HybridClock initialized: now 1790534684739470 us; error 103 us; skew 500 ppm
I20260927 18:44:44.739944 1346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:44 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:44.743336 1346 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:44:44.745028 1346 webserver.cc:460] Webserver started at http://127.31.26.194:42243/ using document root <none> and password file <none>
I20260927 18:44:44.745811 1346 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:44.745966 1346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:44.746291 1346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:44.749423 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "9df85f6ae6714424af21b4b9b5d188ce"
format_stamp: "Formatted at 2026-09-27 18:44:44 on dist-test-slave-rfd5"
server_key: "41bc93e26377d650ac91274eaf8f2e70"
server_key_iv: "e7a98f5f2d2fd66a65a583b77c737462"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.750238 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "9df85f6ae6714424af21b4b9b5d188ce"
format_stamp: "Formatted at 2026-09-27 18:44:44 on dist-test-slave-rfd5"
server_key: "41bc93e26377d650ac91274eaf8f2e70"
server_key_iv: "e7a98f5f2d2fd66a65a583b77c737462"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.755823 1346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:44:44.759923 1361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.761499 1346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:44:44.761711 1346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "9df85f6ae6714424af21b4b9b5d188ce"
format_stamp: "Formatted at 2026-09-27 18:44:44 on dist-test-slave-rfd5"
server_key: "41bc93e26377d650ac91274eaf8f2e70"
server_key_iv: "e7a98f5f2d2fd66a65a583b77c737462"
server_key_version: "encryptionkey@0"
I20260927 18:44:44.761935 1346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:44.789043 1346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:44.793273 1346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:44.793609 1346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:44.796413 1346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:44.796556 1346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:44.796747 1346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:44.796866 1346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:44:44 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534684, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:44.822880 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.797508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:56101 (local address 127.31.26.254:41365)
0927 18:44:44.797692 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:44.797697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:44.799364 (+ 1667us) server_negotiation.cc:407] Connection header received
0927 18:44:44.801626 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:44.801633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:44.801745 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:44.801860 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:44.804685 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.805678 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.808989 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.809482 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.809674 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.814745 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:44.814756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:44.814765 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:44.814821 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:44.818842 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.820156 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.820165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.820171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.820347 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.820944 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.820952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.820957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.821326 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:44.821492 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:44.822316 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 18:44:44.822529 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:44:44.823984 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.798378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.31.26.194:56101)
0927 18:44:44.799199 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.799241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:44.801335 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:44.802327 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:44.802343 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:44.803345 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:44.804458 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.804497 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.805871 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.805880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.808798 (+ 2918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.808823 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.810090 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.810098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.810293 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.811196 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:44.811221 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:44.814467 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:44.819040 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.819053 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.819087 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.819943 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.820498 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.820504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.820512 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.820748 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.821473 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:44.821481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:44.821891 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 18:44:44.822225 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 18:44:44.822776 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":365,"threads_started":1}
I20260927 18:44:44.847784 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.830722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.31.26.194:39739)
0927 18:44:44.831459 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:44.831485 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:44.831672 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:44.832014 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:44.832022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:44.832847 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:44.833399 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.833416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.834575 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.834587 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.836272 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:44.836293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.837934 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.837941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:44.838215 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.839103 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:44.839131 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:44.840669 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:44.844169 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.844176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.844180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.844873 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.845339 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:44.845347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:44.845351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:44.845546 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:44.846315 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:44.846320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:44.846551 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 18:44:44.846762 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 18:44:44.847144 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":358,"threads_started":1}
I20260927 18:44:44.848590 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:44.830776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:39739 (local address 127.31.26.254:41365)
0927 18:44:44.831039 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:44.831049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:44.831590 (+ 541us) server_negotiation.cc:407] Connection header received
0927 18:44:44.831800 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:44.831803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:44.831866 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:44.831946 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:44.833582 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.834417 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.836446 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:44.837432 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:44.837586 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:44.840907 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:44.840912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:44.840916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:44.840948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:44.844032 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.845018 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.845031 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.845035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.845225 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:44.845714 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:44.845719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:44.845723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:44.846198 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:44.846336 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:44.847935 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0927 18:44:44.848243 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:44:44.850750 1346 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:35039
I20260927 18:44:44.850767 1480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:35039 every 8 connection(s)
I20260927 18:44:44.852363 1346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:44.855091 1481 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41365
I20260927 18:44:44.855347 1481 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:44.856009 1481 heartbeater.cc:511] Master 127.31.26.254:41365 requested a full tablet report, sending...
I20260927 18:44:44.857430 1134 ts_manager.cc:194] Registered new tserver with Master: 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:44.858232 1134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:56101
I20260927 18:44:44.861502 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 1346
I20260927 18:44:44.861649 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:44.862041 31851 external_mini_cluster.cc:1428] Setting key 6b96b9c8495dfc7a86bb0d6485a5045a
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:44:44.901678 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41365
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:45.226099 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:45.226438 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:45.226652 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:45.234315 1485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:45.234431 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:45.234611 1485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:44:45.234691 1485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:44:45.234823 1485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:44:45.244951 1485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:41365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:45.246461 1485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:45.247886 1485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:45.262041 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:45.262297 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:45.262146 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:45.262794 1485 server_base.cc:1034] running on GCE node
I20260927 18:44:45.263501 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:45.264544 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:45.265779 1485 hybrid_clock.cc:648] HybridClock initialized: now 1790534685265738 us; error 33 us; skew 500 ppm
I20260927 18:44:45.266124 1485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:45 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534685, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:45.269464 1485 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:44:45.270954 1485 webserver.cc:460] Webserver started at http://127.31.26.195:46829/ using document root <none> and password file <none>
I20260927 18:44:45.271631 1485 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:45.271729 1485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:45.271991 1485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:45.274976 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "4db34bc436044cf3a5a7e4f49e595744"
format_stamp: "Formatted at 2026-09-27 18:44:45 on dist-test-slave-rfd5"
server_key: "2edac8d09a69634308a6121aaa45d573"
server_key_iv: "51861035d807880850b7fd46134e5856"
server_key_version: "encryptionkey@0"
I20260927 18:44:45.275723 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "4db34bc436044cf3a5a7e4f49e595744"
format_stamp: "Formatted at 2026-09-27 18:44:45 on dist-test-slave-rfd5"
server_key: "2edac8d09a69634308a6121aaa45d573"
server_key_iv: "51861035d807880850b7fd46134e5856"
server_key_version: "encryptionkey@0"
I20260927 18:44:45.281175 1485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:44:45.284900 1500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:45.286589 1485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:44:45.286798 1485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "4db34bc436044cf3a5a7e4f49e595744"
format_stamp: "Formatted at 2026-09-27 18:44:45 on dist-test-slave-rfd5"
server_key: "2edac8d09a69634308a6121aaa45d573"
server_key_iv: "51861035d807880850b7fd46134e5856"
server_key_version: "encryptionkey@0"
I20260927 18:44:45.287051 1485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:45.313865 1485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:45.318456 1485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:45.318766 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:45.321537 1485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:45.321630 1485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:45.321802 1485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:45.321892 1485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:44:45 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534685, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:45.339438 1342 heartbeater.cc:503] Master 127.31.26.254:41365 was elected leader, sending a full tablet report...
I20260927 18:44:45.345139 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.322309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:43207 (local address 127.31.26.254:41365)
0927 18:44:45.322484 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.322490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.324183 (+ 1693us) server_negotiation.cc:407] Connection header received
0927 18:44:45.326991 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.327000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.327149 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.327263 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:45.329297 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.330093 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.333354 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.333847 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.333926 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.338317 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:45.338322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:45.338326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:45.338370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:45.341781 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.342674 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.342680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.342684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.342813 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.343266 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.343271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.343274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.343534 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:45.343700 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.344532 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.344750 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:44:45.346339 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.322984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.31.26.195:43207)
0927 18:44:45.323992 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.324044 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.326522 (+ 2478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.327725 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.327743 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.328424 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:45.329018 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.329035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.330252 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.330260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.333142 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.333158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.334231 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.334236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.334419 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.335172 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:45.335198 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:45.337956 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:45.341921 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:45.341930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:45.341948 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:45.342502 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:45.342930 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:45.342933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:45.342935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:45.343141 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:45.343669 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:45.343677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:45.344099 (+ 422us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.344390 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.344994 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":432,"threads_started":1}
I20260927 18:44:45.367727 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.353474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:50135 (local address 127.31.26.254:41365)
0927 18:44:45.353682 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.353690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.354623 (+ 933us) server_negotiation.cc:407] Connection header received
0927 18:44:45.354881 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.354886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.354975 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.355152 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:45.356289 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.356994 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.358529 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.359052 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.359184 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.361881 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:45.361885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:45.361889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:45.361918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:45.364604 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.365495 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.365502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.365505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.365694 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.366155 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.366163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.366167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.366509 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:45.366680 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.367123 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.367346 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:44:45.367919 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.353537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.31.26.195:50135)
0927 18:44:45.354494 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.354519 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.354700 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.355174 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.355179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.355799 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:45.356147 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.356157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.357195 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.357200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.358378 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.358392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.359256 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.359262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.359445 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.360133 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:45.360168 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:45.361670 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:45.364759 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:45.364764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:45.364768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:45.365359 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:45.365881 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:45.365886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:45.365890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:45.366035 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:45.366670 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:45.366675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:45.366886 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.367031 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.367421 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":569,"threads_started":1}
I20260927 18:44:45.369009 1485 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:44701
I20260927 18:44:45.369031 1619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:44701 every 8 connection(s)
I20260927 18:44:45.370293 1485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:45.372570 1620 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41365
I20260927 18:44:45.372795 1620 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:45.373498 1620 heartbeater.cc:511] Master 127.31.26.254:41365 requested a full tablet report, sending...
I20260927 18:44:45.374804 1134 ts_manager.cc:194] Registered new tserver with Master: 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701)
I20260927 18:44:45.375602 1134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:43207
I20260927 18:44:45.379451 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 1485
I20260927 18:44:45.379645 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:45.380091 31851 external_mini_cluster.cc:1428] Setting key 04f0e2fab0434969228c3830806fff59
I20260927 18:44:45.383172 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:44:45 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:44:45 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:45.430878 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.415507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.31.26.254:41365)
0927 18:44:45.415767 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.415773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.416020 (+ 247us) server_negotiation.cc:407] Connection header received
0927 18:44:45.416225 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.416230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.416300 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.416395 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:45.417948 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.418739 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.420015 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.420433 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.420537 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.424717 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:45.424723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:45.424727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:45.424771 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:45.428028 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.428807 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.428812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.428815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.428950 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.429397 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.429403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.429406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.429694 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:45.429949 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.430280 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.430509 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 18:44:45.435406 1134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:45.442541 1134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:45.466156 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.456242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.31.26.194:35039)
0927 18:44:45.457454 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.457466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.458375 (+ 909us) server_negotiation.cc:407] Connection header received
0927 18:44:45.458490 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.458501 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.458789 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.458992 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.459763 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.461309 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.463888 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.464800 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.464966 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.465228 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.465602 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.465786 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":914,"threads_started":1}
I20260927 18:44:45.467151 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.455091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:35039 (local address 127.0.0.1:42834)
0927 18:44:45.458097 (+ 3006us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.458124 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.458238 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.459001 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.459009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.459039 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.459520 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.459538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.461467 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.461471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.463729 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.463741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.465029 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.465035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.465296 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.465342 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.466480 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.466677 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2879,"mutex_wait_us":1,"thread_start_us":2563,"threads_started":1}
I20260927 18:44:45.468197 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.454816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:44701 (local address 127.0.0.1:43436)
0927 18:44:45.456582 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.456634 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.456830 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.460506 (+ 3676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.460510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.460534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.460965 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.460975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.462832 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.462840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.465317 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.465337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.467337 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.467345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.467517 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.467549 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.467664 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.467806 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"mutex_wait_us":178,"thread_start_us":1278,"threads_started":1}
I20260927 18:44:45.468580 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.457158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:45917 (local address 127.0.0.1:56138)
0927 18:44:45.459704 (+ 2546us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.459728 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.459823 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.460808 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.460813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.460841 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.461269 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.461282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.463338 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.463343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.465679 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.465692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.467777 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.467783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.467962 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.467991 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.468080 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.468210 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2405,"mutex_wait_us":33,"thread_start_us":2246,"threads_started":1}
I20260927 18:44:45.468794 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.455147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.31.26.195:44701)
0927 18:44:45.459928 (+ 4781us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.459940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.459981 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:44:45.460090 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.460098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.460323 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.460521 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.461190 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.462701 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.465539 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.466975 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.467156 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.467422 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.468006 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.468204 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4541,"thread_start_us":1547,"threads_started":1}
I20260927 18:44:45.469301 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.457385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.31.26.193:45917)
0927 18:44:45.460281 (+ 2896us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.460290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.460316 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:44:45.460412 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.460419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.460605 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.461498 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.461699 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.463188 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.465922 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.467563 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.467698 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.467888 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.468805 (+ 917us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.468986 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2742,"thread_start_us":2619,"threads_started":1}
I20260927 18:44:45.471256 1411 tablet_service.cc:1467] Processing CreateTablet for tablet 4a17bd19d6cd4e3e9990937c1aca0565 (DEFAULT_TABLE table=test-table [id=c94507a33c3047159e81998e88e62d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:45.472491 1411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a17bd19d6cd4e3e9990937c1aca0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:45.472536 1550 tablet_service.cc:1467] Processing CreateTablet for tablet 4a17bd19d6cd4e3e9990937c1aca0565 (DEFAULT_TABLE table=test-table [id=c94507a33c3047159e81998e88e62d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:45.472536 1272 tablet_service.cc:1467] Processing CreateTablet for tablet 4a17bd19d6cd4e3e9990937c1aca0565 (DEFAULT_TABLE table=test-table [id=c94507a33c3047159e81998e88e62d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:45.473886 1272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a17bd19d6cd4e3e9990937c1aca0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:45.473896 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a17bd19d6cd4e3e9990937c1aca0565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:45.484038 1639 tablet_bootstrap.cc:492] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Bootstrap starting.
I20260927 18:44:45.487099 1639 tablet_bootstrap.cc:654] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:45.488237 1640 tablet_bootstrap.cc:492] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Bootstrap starting.
I20260927 18:44:45.488405 1639 log.cc:824] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:45.489928 1641 tablet_bootstrap.cc:492] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Bootstrap starting.
I20260927 18:44:45.492959 1640 tablet_bootstrap.cc:654] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:45.493381 1641 tablet_bootstrap.cc:654] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:45.494349 1641 log.cc:824] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:45.494601 1640 log.cc:824] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:45.497503 1641 tablet_bootstrap.cc:492] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: No bootstrap required, opened a new log
I20260927 18:44:45.497668 1639 tablet_bootstrap.cc:492] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: No bootstrap required, opened a new log
I20260927 18:44:45.497668 1640 tablet_bootstrap.cc:492] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: No bootstrap required, opened a new log
I20260927 18:44:45.497815 1641 ts_tablet_manager.cc:1397] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260927 18:44:45.497942 1640 ts_tablet_manager.cc:1397] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:44:45.497931 1639 ts_tablet_manager.cc:1397] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260927 18:44:45.505321 1641 raft_consensus.cc:348] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.505719 1641 raft_consensus.cc:374] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:45.505812 1641 raft_consensus.cc:729] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Initialized, Role: FOLLOWER
I20260927 18:44:45.506640 1641 consensus_queue.cc:260] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.508075 1620 heartbeater.cc:503] Master 127.31.26.254:41365 was elected leader, sending a full tablet report...
I20260927 18:44:45.508817 1641 ts_tablet_manager.cc:1428] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260927 18:44:45.508649 1639 raft_consensus.cc:348] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.508651 1640 raft_consensus.cc:348] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.509069 1639 raft_consensus.cc:374] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:45.509069 1640 raft_consensus.cc:374] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:45.509200 1639 raft_consensus.cc:729] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:45.509200 1640 raft_consensus.cc:729] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cda65bc8042dbb8967f5032022eaf, State: Initialized, Role: FOLLOWER
I20260927 18:44:45.509830 1639 consensus_queue.cc:260] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.509833 1640 consensus_queue.cc:260] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.512125 1640 ts_tablet_manager.cc:1428] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260927 18:44:45.512357 1639 ts_tablet_manager.cc:1428] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 18:44:45.512579 1481 heartbeater.cc:503] Master 127.31.26.254:41365 was elected leader, sending a full tablet report...
W20260927 18:44:45.579944 1343 tablet.cc:2369] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:45.604773 1482 tablet.cc:2369] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:44:45.622761 1621 tablet.cc:2369] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:45.626421 1646 raft_consensus.cc:482] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:45.626793 1646 raft_consensus.cc:504] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.628779 1646 leader_election.cc:288] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701), 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917)
I20260927 18:44:45.635922 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.629356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:46641 (local address 127.31.26.193:45917)
0927 18:44:45.629560 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.629565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.630422 (+ 857us) server_negotiation.cc:407] Connection header received
0927 18:44:45.630664 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.630673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.630732 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.630827 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.631368 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.632277 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.634194 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.634954 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.635141 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.635226 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.635530 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.635654 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:44:45.635950 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.629346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:59747 (local address 127.31.26.195:44701)
0927 18:44:45.629563 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.629572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.630677 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 18:44:45.630891 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.630896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.630952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.631049 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.631624 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.632301 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.634221 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.635001 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.635192 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.635266 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.635550 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.635673 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:44:45.636103 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.629326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:44701 (local address 127.31.26.194:59747)
0927 18:44:45.630565 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.630580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.630737 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.631066 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.631070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.631093 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.631479 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.631487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.632439 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.632445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.634052 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.634063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.635222 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.635228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.635398 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.635421 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.635524 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.635652 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":23,"thread_start_us":493,"threads_started":1}
I20260927 18:44:45.636263 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.629293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:45917 (local address 127.31.26.194:46641)
0927 18:44:45.630280 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.630301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.630513 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.630854 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.630858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.630882 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.631223 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.631233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.632402 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.632411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.634026 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.634038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.635188 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.635194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.635366 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.635391 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.635506 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.635647 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":3,"thread_start_us":916,"threads_started":1}
I20260927 18:44:45.637533 1573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a17bd19d6cd4e3e9990937c1aca0565" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db34bc436044cf3a5a7e4f49e595744" is_pre_election: true
I20260927 18:44:45.637511 1292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a17bd19d6cd4e3e9990937c1aca0565" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf" is_pre_election: true
I20260927 18:44:45.638134 1292 raft_consensus.cc:2454] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 0.
I20260927 18:44:45.638134 1573 raft_consensus.cc:2454] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 0.
I20260927 18:44:45.638921 1363 leader_election.cc:302] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:45.639397 1646 raft_consensus.cc:2781] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:45.639564 1646 raft_consensus.cc:482] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:45.639703 1646 raft_consensus.cc:3037] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:45.641997 1646 raft_consensus.cc:504] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.642644 1646 leader_election.cc:288] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 election: Requested vote from peers 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701), 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917)
I20260927 18:44:45.643226 1292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a17bd19d6cd4e3e9990937c1aca0565" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf"
I20260927 18:44:45.643226 1573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a17bd19d6cd4e3e9990937c1aca0565" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db34bc436044cf3a5a7e4f49e595744"
I20260927 18:44:45.643451 1292 raft_consensus.cc:3037] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:45.643447 1573 raft_consensus.cc:3037] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:45.646737 1292 raft_consensus.cc:2454] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 1.
I20260927 18:44:45.646737 1573 raft_consensus.cc:2454] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 1.
I20260927 18:44:45.647265 1364 leader_election.cc:302] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 905cda65bc8042dbb8967f5032022eaf, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:45.647686 1646 raft_consensus.cc:2781] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:45.648180 1646 raft_consensus.cc:686] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 LEADER]: Becoming Leader. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Running, Role: LEADER
I20260927 18:44:45.648753 1646 consensus_queue.cc:237] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } }
I20260927 18:44:45.653718 1134 catalog_manager.cc:5495] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194). New cstate: current_term: 1 leader_uuid: "9df85f6ae6714424af21b4b9b5d188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:44:45 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534683, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:45.744808 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.731604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41365 (local address 127.0.0.1:45318)
0927 18:44:45.732235 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.732251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.732385 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.732704 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.732707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.733483 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:45.733743 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.733752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.734748 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.734751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.735698 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.735709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.736431 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.736436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.736554 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.737364 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:45.737387 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:45.739664 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:45.742574 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:45.742587 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:45.742593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:45.742982 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:45.743300 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:45.743303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:45.743306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:45.743428 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:45.743906 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:45.743913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:45.744064 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.744207 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.744443 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":354,"threads_started":1}
I20260927 18:44:45.744812 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.731594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.31.26.254:41365)
0927 18:44:45.731770 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.731778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.732349 (+ 571us) server_negotiation.cc:407] Connection header received
0927 18:44:45.732491 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.732495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.732568 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.732654 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:45.733888 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.734632 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.735833 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.736260 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.736362 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.739843 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:45.739849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:45.739852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:45.739881 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:45.742442 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.743083 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.743087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.743089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.743207 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:45.743536 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:45.743539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:45.743541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:45.743800 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:45.743906 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.744329 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.744553 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":60}
I20260927 18:44:45.750054 1134 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:45318:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:44:45.751077 1134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:45.758023 1272 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3e7894455b4338aaa2186221c84aae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2fbb62714f6848d58dfa51b396280dd0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:44:45.758312 1411 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3e7894455b4338aaa2186221c84aae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2fbb62714f6848d58dfa51b396280dd0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:44:45.759084 1272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3e7894455b4338aaa2186221c84aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:45.759084 1411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3e7894455b4338aaa2186221c84aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:45.761580 1550 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3e7894455b4338aaa2186221c84aae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2fbb62714f6848d58dfa51b396280dd0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:44:45.762454 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3e7894455b4338aaa2186221c84aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:45.766467 1639 tablet_bootstrap.cc:492] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: Bootstrap starting.
I20260927 18:44:45.766875 1640 tablet_bootstrap.cc:492] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: Bootstrap starting.
I20260927 18:44:45.768151 1640 tablet_bootstrap.cc:654] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:45.768355 1639 tablet_bootstrap.cc:654] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:45.768733 1641 tablet_bootstrap.cc:492] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: Bootstrap starting.
I20260927 18:44:45.770844 1641 tablet_bootstrap.cc:654] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:45.771975 1640 tablet_bootstrap.cc:492] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: No bootstrap required, opened a new log
I20260927 18:44:45.772034 1639 tablet_bootstrap.cc:492] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: No bootstrap required, opened a new log
I20260927 18:44:45.772161 1640 ts_tablet_manager.cc:1397] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:45.772214 1639 ts_tablet_manager.cc:1397] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:44:45.772903 1639 raft_consensus.cc:348] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.773207 1639 raft_consensus.cc:374] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:45.773317 1639 raft_consensus.cc:729] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:45.773128 1640 raft_consensus.cc:348] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.773633 1640 raft_consensus.cc:374] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:45.773710 1641 tablet_bootstrap.cc:492] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: No bootstrap required, opened a new log
I20260927 18:44:45.773558 1639 consensus_queue.cc:260] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.773831 1641 ts_tablet_manager.cc:1397] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:45.773996 1640 raft_consensus.cc:729] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cda65bc8042dbb8967f5032022eaf, State: Initialized, Role: FOLLOWER
I20260927 18:44:45.774154 1639 ts_tablet_manager.cc:1428] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:44:45.774597 1640 consensus_queue.cc:260] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.774605 1641 raft_consensus.cc:348] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.774838 1641 raft_consensus.cc:374] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:45.774983 1646 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.775259 1646 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.775106 1647 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.775528 1647 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.775467 1641 raft_consensus.cc:729] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Initialized, Role: FOLLOWER
I20260927 18:44:45.776237 1640 ts_tablet_manager.cc:1428] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:45.776367 1641 consensus_queue.cc:260] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.776889 1645 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.777359 1645 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.778367 1641 ts_tablet_manager.cc:1428] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 18:44:45.831664 1645 raft_consensus.cc:482] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:45.831931 1645 raft_consensus.cc:504] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.833571 1645 leader_election.cc:288] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:45.840133 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.833889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:45917 (local address 127.31.26.195:58445)
0927 18:44:45.834180 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.834211 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.834399 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.834812 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.834816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.834846 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.835059 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.835070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.836258 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.836264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.838086 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.838098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.839353 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.839360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.839580 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.839616 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.839719 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.839865 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":57}
I20260927 18:44:45.840204 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.833958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:58445 (local address 127.31.26.193:45917)
0927 18:44:45.834232 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.834242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.834293 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:44:45.834561 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.834568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.834651 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.834747 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.835208 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.836127 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.838251 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.839190 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.839319 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.839371 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.839759 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.839898 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 18:44:45.840505 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.833925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:54169 (local address 127.31.26.194:35039)
0927 18:44:45.834128 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.834135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.834778 (+ 643us) server_negotiation.cc:407] Connection header received
0927 18:44:45.834859 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.834864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.834934 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.835068 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:44:45.835676 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.836519 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.838533 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.839481 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.839645 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.839732 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.840100 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.840245 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:44:45.840535 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.833939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:35039 (local address 127.31.26.195:54169)
0927 18:44:45.834614 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.834632 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.834763 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.835079 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.835082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.835101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:44:45.835505 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.835517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.836705 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.836709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.838328 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.838344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.839720 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.839727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.839924 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.839958 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.840051 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.840175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":5,"thread_start_us":337,"threads_started":1}
I20260927 18:44:45.841459 1292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3e7894455b4338aaa2186221c84aae" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf" is_pre_election: true
I20260927 18:44:45.841727 1292 raft_consensus.cc:2454] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 0.
I20260927 18:44:45.841774 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3e7894455b4338aaa2186221c84aae" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df85f6ae6714424af21b4b9b5d188ce" is_pre_election: true
I20260927 18:44:45.842226 1503 leader_election.cc:302] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 905cda65bc8042dbb8967f5032022eaf; no voters:
I20260927 18:44:45.842242 1436 raft_consensus.cc:2454] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 0.
I20260927 18:44:45.842588 1645 raft_consensus.cc:2781] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:45.842725 1645 raft_consensus.cc:482] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:45.842841 1645 raft_consensus.cc:3037] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:45.844920 1645 raft_consensus.cc:504] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.845551 1645 leader_election.cc:288] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 election: Requested vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:45.845968 1292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3e7894455b4338aaa2186221c84aae" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf"
I20260927 18:44:45.846035 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3e7894455b4338aaa2186221c84aae" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df85f6ae6714424af21b4b9b5d188ce"
I20260927 18:44:45.846185 1292 raft_consensus.cc:3037] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:45.846244 1436 raft_consensus.cc:3037] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:45.848297 1292 raft_consensus.cc:2454] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 1.
I20260927 18:44:45.848402 1436 raft_consensus.cc:2454] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 1.
I20260927 18:44:45.848752 1503 leader_election.cc:302] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 905cda65bc8042dbb8967f5032022eaf; no voters:
I20260927 18:44:45.849141 1645 raft_consensus.cc:2781] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:45.849591 1645 raft_consensus.cc:686] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [term 1 LEADER]: Becoming Leader. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Running, Role: LEADER
I20260927 18:44:45.850157 1645 consensus_queue.cc:237] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:45.853734 1662 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: TxnStatusTablet state changed. Reason: New leader 4db34bc436044cf3a5a7e4f49e595744. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.854137 1662 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:44:45.854882 1664 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:44:45.855147 1664 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:44:45.855629 1134 catalog_manager.cc:5495] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195). New cstate: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260927 18:44:45.875403 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.869505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.31.26.195:44701)
0927 18:44:45.869681 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.869687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.869983 (+ 296us) server_negotiation.cc:407] Connection header received
0927 18:44:45.870290 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.870295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.870382 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.870529 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:45.870895 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.871694 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.873165 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.873584 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.873741 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.874213 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.874721 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.874889 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.874974 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.875060 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:44:45.875393 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.869532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:44701 (local address 127.0.0.1:43440)
0927 18:44:45.869887 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.869903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.870068 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.870555 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.870559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.870572 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:45.870758 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.870767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.871825 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.871832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.872999 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.873011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.873761 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.873766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.873959 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.874012 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.874852 (+ 840us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.874871 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.874955 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.875106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260927 18:44:45.883389 1436 raft_consensus.cc:1261] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Refusing update from remote peer 4db34bc436044cf3a5a7e4f49e595744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:45.883594 1292 raft_consensus.cc:1261] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Refusing update from remote peer 4db34bc436044cf3a5a7e4f49e595744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:45.884343 1645 consensus_queue.cc:1035] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:45.884665 1662 consensus_queue.cc:1035] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:45.892150 1646 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: TxnStatusTablet state changed. Reason: New leader 4db34bc436044cf3a5a7e4f49e595744. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.892521 1646 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.894261 1667 mvcc.cc:205] Tried to move back new op lower bound from 7334030073368723456 to 7334030073246064640. Current Snapshot: MvccSnapshot[applied={T|T < 7334030073368723456}]
I20260927 18:44:45.900189 1645 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.900482 1645 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:44:45.905541 1647 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: TxnStatusTablet state changed. Reason: New leader 4db34bc436044cf3a5a7e4f49e595744. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.905855 1647 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.915127 1647 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.915421 1647 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 905cda65bc8042dbb8967f5032022eaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.918852 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.911185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.31.26.194:35039)
0927 18:44:45.911353 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.911358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.911375 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:44:45.911471 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.911478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.911584 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.911730 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:45.912313 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.913475 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.915521 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.916183 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.916316 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.916977 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.917695 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.917856 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.918452 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.918539 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:44:45.920607 1646 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.921020 1646 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 9df85f6ae6714424af21b4b9b5d188ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:44:45.925617 1662 tablet_replica.cc:439] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } } }
I20260927 18:44:45.925935 1662 tablet_replica.cc:442] T 3b3e7894455b4338aaa2186221c84aae P 4db34bc436044cf3a5a7e4f49e595744: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:44:45.938858 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.932061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:55105 (local address 127.31.26.195:44701)
0927 18:44:45.932328 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.932337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.932849 (+ 512us) server_negotiation.cc:407] Connection header received
0927 18:44:45.933058 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.933062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.933187 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.933263 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:45.933875 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.934607 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.936697 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.937394 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.937491 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.938049 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.938209 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.938293 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.938501 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.938588 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:44:45.939004 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.932051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:44701 (local address 127.31.26.194:55105)
0927 18:44:45.932735 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.932755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.932916 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.933311 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.933316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.933330 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:45.933707 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.933721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.934741 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.934749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.936520 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.936536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.937558 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.937565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.937743 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.937821 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.938325 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.938356 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.938444 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.938608 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":353,"threads_started":1}
I20260927 18:44:45.950387 1291 raft_consensus.cc:1261] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Refusing update from remote peer 9df85f6ae6714424af21b4b9b5d188ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:45.950608 1573 raft_consensus.cc:1261] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Refusing update from remote peer 9df85f6ae6714424af21b4b9b5d188ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:45.951079 1650 consensus_queue.cc:1035] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:45.951469 1646 consensus_queue.cc:1035] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:45.957754 1668 mvcc.cc:205] Tried to move back new op lower bound from 7334030073645891584 to 7334030072421584896. Current Snapshot: MvccSnapshot[applied={T|T < 7334030073645891584}]
I20260927 18:44:45.957981 1669 mvcc.cc:205] Tried to move back new op lower bound from 7334030073645891584 to 7334030072421584896. Current Snapshot: MvccSnapshot[applied={T|T < 7334030073645891584}]
I20260927 18:44:45.999725 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.984040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:56125 (local address 127.31.26.194:35039)
0927 18:44:45.984217 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:45.984224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:45.993815 (+ 9591us) server_negotiation.cc:407] Connection header received
0927 18:44:45.994011 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:45.994017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:45.994125 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:45.994224 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:45.994986 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.995711 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.997854 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.998437 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.998577 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.999081 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.999246 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.999339 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:45.999453 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 18:44:45.999523 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:44:46.000296 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:45.984040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:35039 (local address 127.31.26.195:56125)
0927 18:44:45.993687 (+ 9647us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:45.993710 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:45.993878 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:45.994399 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:45.994405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:45.994418 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:45.994828 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.994839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.995881 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.995887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.997708 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:45.997723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:45.998631 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:45.998637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:45.998825 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:45.998899 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:45.999343 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:45.999355 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:44:45.999440 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:44:45.999595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9409,"thread_start_us":9253,"threads_started":1}
I20260927 18:44:46.056982 1135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:46.057312 1135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:46.061463 1135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Sending DeleteTablet for 3 replicas of tablet 4a17bd19d6cd4e3e9990937c1aca0565
I20260927 18:44:46.063009 1550 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a17bd19d6cd4e3e9990937c1aca0565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:46 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:43436
I20260927 18:44:46.063203 1272 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a17bd19d6cd4e3e9990937c1aca0565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:46 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:56138
I20260927 18:44:46.064065 1411 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a17bd19d6cd4e3e9990937c1aca0565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:46 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:42834
I20260927 18:44:46.067935 1135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:46.068679 1135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:46.076237 1410 tablet_service.cc:1467] Processing CreateTablet for tablet b9b4d14d9b1a4ab486920e536d67dae5 (DEFAULT_TABLE table=test-table [id=63317001d75d45cd93b2caf33b0ab090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:46.076835 1410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b4d14d9b1a4ab486920e536d67dae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:46.082489 1549 tablet_service.cc:1467] Processing CreateTablet for tablet b9b4d14d9b1a4ab486920e536d67dae5 (DEFAULT_TABLE table=test-table [id=63317001d75d45cd93b2caf33b0ab090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:46.083177 1549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b4d14d9b1a4ab486920e536d67dae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:46.083273 1689 tablet_replica.cc:330] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: stopping tablet replica
I20260927 18:44:46.083729 1689 raft_consensus.cc:2229] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:46.084105 1689 raft_consensus.cc:2258] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:46.086083 1689 ts_tablet_manager.cc:1905] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:46.089593 1271 tablet_service.cc:1467] Processing CreateTablet for tablet b9b4d14d9b1a4ab486920e536d67dae5 (DEFAULT_TABLE table=test-table [id=63317001d75d45cd93b2caf33b0ab090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:46.090097 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b4d14d9b1a4ab486920e536d67dae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:46.093662 1689 ts_tablet_manager.cc:1918] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:46.093849 1689 log.cc:1196] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/4a17bd19d6cd4e3e9990937c1aca0565
I20260927 18:44:46.094424 1689 ts_tablet_manager.cc:1939] T 4a17bd19d6cd4e3e9990937c1aca0565 P 4db34bc436044cf3a5a7e4f49e595744: Deleting consensus metadata
I20260927 18:44:46.095486 1639 tablet_bootstrap.cc:492] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Bootstrap starting.
I20260927 18:44:46.096228 1120 catalog_manager.cc:4841] TS 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701): tablet 4a17bd19d6cd4e3e9990937c1aca0565 (table test-table [id=c94507a33c3047159e81998e88e62d44]) successfully deleted
I20260927 18:44:46.097286 1639 tablet_bootstrap.cc:654] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:46.097442 1641 tablet_bootstrap.cc:492] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Bootstrap starting.
I20260927 18:44:46.097868 1640 tablet_bootstrap.cc:492] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Bootstrap starting.
I20260927 18:44:46.099428 1640 tablet_bootstrap.cc:654] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:46.099428 1641 tablet_bootstrap.cc:654] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:46.118845 1641 tablet_bootstrap.cc:492] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: No bootstrap required, opened a new log
I20260927 18:44:46.119093 1641 ts_tablet_manager.cc:1397] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20260927 18:44:46.120087 1641 raft_consensus.cc:348] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.120375 1641 raft_consensus.cc:374] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:46.120452 1641 raft_consensus.cc:729] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Initialized, Role: FOLLOWER
I20260927 18:44:46.120752 1641 consensus_queue.cc:260] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.122838 1691 tablet_replica.cc:330] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: stopping tablet replica
I20260927 18:44:46.123497 1691 raft_consensus.cc:2229] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:46.124351 1691 raft_consensus.cc:2258] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:46.125219 1690 tablet_replica.cc:330] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: stopping tablet replica
I20260927 18:44:46.125669 1690 raft_consensus.cc:2229] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:46.126075 1690 raft_consensus.cc:2258] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:46.126559 1641 ts_tablet_manager.cc:1428] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260927 18:44:46.128258 1639 tablet_bootstrap.cc:492] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: No bootstrap required, opened a new log
I20260927 18:44:46.128423 1639 ts_tablet_manager.cc:1397] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.009s
I20260927 18:44:46.128481 1690 ts_tablet_manager.cc:1905] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:46.129374 1639 raft_consensus.cc:348] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.129642 1639 raft_consensus.cc:374] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:46.129727 1639 raft_consensus.cc:729] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:46.129998 1639 consensus_queue.cc:260] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.130391 1640 tablet_bootstrap.cc:492] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: No bootstrap required, opened a new log
I20260927 18:44:46.130983 1640 ts_tablet_manager.cc:1397] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.011s
I20260927 18:44:46.131810 1640 raft_consensus.cc:348] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.132117 1640 raft_consensus.cc:374] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:46.132241 1640 raft_consensus.cc:729] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cda65bc8042dbb8967f5032022eaf, State: Initialized, Role: FOLLOWER
I20260927 18:44:46.132599 1640 consensus_queue.cc:260] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.133826 1639 ts_tablet_manager.cc:1428] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:44:46.135509 1640 ts_tablet_manager.cc:1428] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 18:44:46.136183 1690 ts_tablet_manager.cc:1918] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:46.136875 1691 ts_tablet_manager.cc:1905] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:46.144881 1691 ts_tablet_manager.cc:1918] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:46.145133 1691 log.cc:1196] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/4a17bd19d6cd4e3e9990937c1aca0565
I20260927 18:44:46.145309 1690 log.cc:1196] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/4a17bd19d6cd4e3e9990937c1aca0565
I20260927 18:44:46.145705 1691 ts_tablet_manager.cc:1939] T 4a17bd19d6cd4e3e9990937c1aca0565 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting consensus metadata
I20260927 18:44:46.146173 1690 ts_tablet_manager.cc:1939] T 4a17bd19d6cd4e3e9990937c1aca0565 P 905cda65bc8042dbb8967f5032022eaf: Deleting consensus metadata
I20260927 18:44:46.147835 1119 catalog_manager.cc:4841] TS 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039): tablet 4a17bd19d6cd4e3e9990937c1aca0565 (table test-table [id=c94507a33c3047159e81998e88e62d44]) successfully deleted
I20260927 18:44:46.147902 1121 catalog_manager.cc:4841] TS 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917): tablet 4a17bd19d6cd4e3e9990937c1aca0565 (table test-table [id=c94507a33c3047159e81998e88e62d44]) successfully deleted
I20260927 18:44:46.180106 1680 raft_consensus.cc:482] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:46.180353 1680 raft_consensus.cc:504] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.181360 1680 leader_election.cc:288] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:46.181981 1292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b4d14d9b1a4ab486920e536d67dae5" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf" is_pre_election: true
I20260927 18:44:46.182027 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b4d14d9b1a4ab486920e536d67dae5" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df85f6ae6714424af21b4b9b5d188ce" is_pre_election: true
I20260927 18:44:46.182262 1292 raft_consensus.cc:2454] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 0.
I20260927 18:44:46.182309 1436 raft_consensus.cc:2454] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 0.
I20260927 18:44:46.182797 1503 leader_election.cc:302] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 905cda65bc8042dbb8967f5032022eaf; no voters:
I20260927 18:44:46.183125 1680 raft_consensus.cc:2781] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:46.183264 1680 raft_consensus.cc:482] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:46.183367 1680 raft_consensus.cc:3037] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:46.186844 1680 raft_consensus.cc:504] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.187618 1680 leader_election.cc:288] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 election: Requested vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:46.188335 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b4d14d9b1a4ab486920e536d67dae5" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df85f6ae6714424af21b4b9b5d188ce"
I20260927 18:44:46.188802 1436 raft_consensus.cc:3037] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:46.188225 1292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b4d14d9b1a4ab486920e536d67dae5" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf"
I20260927 18:44:46.189692 1292 raft_consensus.cc:3037] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:46.192390 1436 raft_consensus.cc:2454] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 1.
I20260927 18:44:46.192602 1292 raft_consensus.cc:2454] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 1.
I20260927 18:44:46.192927 1501 leader_election.cc:302] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:46.193792 1680 raft_consensus.cc:2781] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:46.194005 1680 raft_consensus.cc:686] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 LEADER]: Becoming Leader. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Running, Role: LEADER
I20260927 18:44:46.194358 1680 consensus_queue.cc:237] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:46.197521 1135 catalog_manager.cc:5495] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195). New cstate: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260927 18:44:46.242086 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.235458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.31.26.195:44701)
0927 18:44:46.235651 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:46.235657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:46.235677 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:44:46.235814 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:46.235819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:46.235902 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:46.235999 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:46.236410 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.237730 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.239787 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.240527 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.240662 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.241212 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:46.241379 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:46.241469 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:46.241652 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:44:46.241737 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:44:46.260355 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.253873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:55889 (local address 127.31.26.195:44701)
0927 18:44:46.254053 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:46.254059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:46.254089 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:44:46.254148 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:46.254153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:46.254230 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:46.254342 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:46.254881 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.256118 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.258215 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.258867 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.259003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.259491 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:46.259621 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:46.259720 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:46.259874 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 18:44:46.259962 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":55}
I20260927 18:44:46.260381 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:46.253480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:44701 (local address 127.31.26.195:55889)
0927 18:44:46.253712 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:46.253736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:46.253879 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:46.254386 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:46.254392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:46.254408 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:44:46.254728 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:46.254742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.256286 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.256292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:46.258025 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:46.258045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:46.259039 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:46.259045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:46.259236 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:46.259286 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:44:46.259732 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:44:46.259743 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:44:46.259830 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:44:46.259987 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":55}
I20260927 18:44:46.271804 1435 raft_consensus.cc:1261] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Refusing update from remote peer 4db34bc436044cf3a5a7e4f49e595744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:46.271982 1292 raft_consensus.cc:1261] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Refusing update from remote peer 4db34bc436044cf3a5a7e4f49e595744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:46.272392 1662 consensus_queue.cc:1035] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:46.272855 1680 consensus_queue.cc:1035] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:44:46.313584 1667 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:44:47.929462 1134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:47.929729 1134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:47.936760 1134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Sending DeleteTablet for 3 replicas of tablet b9b4d14d9b1a4ab486920e536d67dae5
I20260927 18:44:47.937649 1272 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b4d14d9b1a4ab486920e536d67dae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:47 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:56138
I20260927 18:44:47.937670 1411 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b4d14d9b1a4ab486920e536d67dae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:47 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:42834
I20260927 18:44:47.938131 1549 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b4d14d9b1a4ab486920e536d67dae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:47 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:43436
I20260927 18:44:47.938328 1718 tablet_replica.cc:330] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: stopping tablet replica
I20260927 18:44:47.938455 1719 tablet_replica.cc:330] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: stopping tablet replica
I20260927 18:44:47.938545 1718 raft_consensus.cc:2229] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:47.938827 1718 raft_consensus.cc:2258] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:47.939397 1719 raft_consensus.cc:2229] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:47.939782 1719 raft_consensus.cc:2258] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:47.939843 1718 ts_tablet_manager.cc:1905] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:47.941126 1719 ts_tablet_manager.cc:1905] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:47.947469 1718 ts_tablet_manager.cc:1918] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:44:47.947628 1718 log.cc:1196] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/b9b4d14d9b1a4ab486920e536d67dae5
I20260927 18:44:47.948148 1718 ts_tablet_manager.cc:1939] T b9b4d14d9b1a4ab486920e536d67dae5 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting consensus metadata
I20260927 18:44:47.948192 1720 tablet_replica.cc:330] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: stopping tablet replica
I20260927 18:44:47.948416 1720 raft_consensus.cc:2229] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:47.949110 1720 raft_consensus.cc:2258] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:47.949399 1119 catalog_manager.cc:4841] TS 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039): tablet b9b4d14d9b1a4ab486920e536d67dae5 (table test-table [id=63317001d75d45cd93b2caf33b0ab090]) successfully deleted
I20260927 18:44:47.950234 1719 ts_tablet_manager.cc:1918] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:44:47.950430 1719 log.cc:1196] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/b9b4d14d9b1a4ab486920e536d67dae5
I20260927 18:44:47.950981 1719 ts_tablet_manager.cc:1939] T b9b4d14d9b1a4ab486920e536d67dae5 P 905cda65bc8042dbb8967f5032022eaf: Deleting consensus metadata
I20260927 18:44:47.952348 1720 ts_tablet_manager.cc:1905] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:47.953574 1121 catalog_manager.cc:4841] TS 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917): tablet b9b4d14d9b1a4ab486920e536d67dae5 (table test-table [id=63317001d75d45cd93b2caf33b0ab090]) successfully deleted
I20260927 18:44:47.959692 1720 ts_tablet_manager.cc:1918] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:44:47.959854 1720 log.cc:1196] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/b9b4d14d9b1a4ab486920e536d67dae5
I20260927 18:44:47.960361 1720 ts_tablet_manager.cc:1939] T b9b4d14d9b1a4ab486920e536d67dae5 P 4db34bc436044cf3a5a7e4f49e595744: Deleting consensus metadata
I20260927 18:44:47.961556 1120 catalog_manager.cc:4841] TS 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701): tablet b9b4d14d9b1a4ab486920e536d67dae5 (table test-table [id=63317001d75d45cd93b2caf33b0ab090]) successfully deleted
Sep 27 18:44:47 dist-test-slave-rfd5 krb5kdc[1090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:44:47 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:47.996178 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:47.981538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.31.26.254:41365)
0927 18:44:47.982080 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:47.982084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:47.982169 (+ 85us) server_negotiation.cc:407] Connection header received
0927 18:44:47.982318 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:47.982322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:47.982380 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:47.982481 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:47.983725 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:47.984491 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:47.985800 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:47.986223 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:47.986363 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:47.990487 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:47.990492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:47.990495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:47.990602 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:47.993745 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:47.994488 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:47.994492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:47.994495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:47.994632 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:47.994976 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:47.994979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:47.994981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:47.995262 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:47.995438 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:47.995750 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 18:44:47.995920 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":276,"threads_started":1}
I20260927 18:44:48.000075 1134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:48.000708 1134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:48.007812 1272 tablet_service.cc:1467] Processing CreateTablet for tablet 557bda0fc43a46268d025443064e2a49 (DEFAULT_TABLE table=test-table [id=64aa1e701cbc42b6bd2c26ae8f8626ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:48.008031 1549 tablet_service.cc:1467] Processing CreateTablet for tablet 557bda0fc43a46268d025443064e2a49 (DEFAULT_TABLE table=test-table [id=64aa1e701cbc42b6bd2c26ae8f8626ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:48.008128 1411 tablet_service.cc:1467] Processing CreateTablet for tablet 557bda0fc43a46268d025443064e2a49 (DEFAULT_TABLE table=test-table [id=64aa1e701cbc42b6bd2c26ae8f8626ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:48.008298 1272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557bda0fc43a46268d025443064e2a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:48.008471 1549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557bda0fc43a46268d025443064e2a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:48.008505 1411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557bda0fc43a46268d025443064e2a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:48.014098 1729 tablet_bootstrap.cc:492] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Bootstrap starting.
I20260927 18:44:48.016492 1729 tablet_bootstrap.cc:654] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:48.017673 1730 tablet_bootstrap.cc:492] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Bootstrap starting.
I20260927 18:44:48.020088 1730 tablet_bootstrap.cc:654] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:48.021842 1731 tablet_bootstrap.cc:492] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Bootstrap starting.
I20260927 18:44:48.022567 1729 tablet_bootstrap.cc:492] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: No bootstrap required, opened a new log
I20260927 18:44:48.022778 1729 ts_tablet_manager.cc:1397] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 18:44:48.023605 1731 tablet_bootstrap.cc:654] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:48.023610 1729 raft_consensus.cc:348] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.023854 1729 raft_consensus.cc:374] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:48.023952 1729 raft_consensus.cc:729] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cda65bc8042dbb8967f5032022eaf, State: Initialized, Role: FOLLOWER
I20260927 18:44:48.024623 1729 consensus_queue.cc:260] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.025624 1729 ts_tablet_manager.cc:1428] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 18:44:48.026126 1730 tablet_bootstrap.cc:492] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: No bootstrap required, opened a new log
I20260927 18:44:48.026325 1730 ts_tablet_manager.cc:1397] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 18:44:48.026957 1731 tablet_bootstrap.cc:492] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: No bootstrap required, opened a new log
I20260927 18:44:48.027058 1731 ts_tablet_manager.cc:1397] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:48.027144 1730 raft_consensus.cc:348] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.027377 1730 raft_consensus.cc:374] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:48.027474 1730 raft_consensus.cc:729] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Initialized, Role: FOLLOWER
I20260927 18:44:48.027688 1730 consensus_queue.cc:260] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.027772 1731 raft_consensus.cc:348] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.028376 1730 ts_tablet_manager.cc:1428] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:48.028844 1731 raft_consensus.cc:374] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:48.028972 1731 raft_consensus.cc:729] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:48.029280 1731 consensus_queue.cc:260] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.033761 1731 ts_tablet_manager.cc:1428] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 18:44:48.124099 1736 raft_consensus.cc:482] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:48.124327 1736 raft_consensus.cc:504] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.125003 1736 leader_election.cc:288] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701)
I20260927 18:44:48.125628 1573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557bda0fc43a46268d025443064e2a49" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db34bc436044cf3a5a7e4f49e595744" is_pre_election: true
I20260927 18:44:48.125658 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557bda0fc43a46268d025443064e2a49" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf" is_pre_election: true
I20260927 18:44:48.125876 1573 raft_consensus.cc:2454] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 0.
I20260927 18:44:48.125916 1291 raft_consensus.cc:2454] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 0.
I20260927 18:44:48.126392 1363 leader_election.cc:302] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:48.126701 1736 raft_consensus.cc:2781] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:48.126814 1736 raft_consensus.cc:482] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:48.126878 1736 raft_consensus.cc:3037] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:48.129050 1736 raft_consensus.cc:504] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.129714 1736 leader_election.cc:288] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 election: Requested vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701)
I20260927 18:44:48.130198 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557bda0fc43a46268d025443064e2a49" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf"
I20260927 18:44:48.130260 1573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557bda0fc43a46268d025443064e2a49" candidate_uuid: "9df85f6ae6714424af21b4b9b5d188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db34bc436044cf3a5a7e4f49e595744"
I20260927 18:44:48.130434 1291 raft_consensus.cc:3037] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:48.130486 1573 raft_consensus.cc:3037] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:48.132839 1573 raft_consensus.cc:2454] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 1.
I20260927 18:44:48.132988 1291 raft_consensus.cc:2454] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df85f6ae6714424af21b4b9b5d188ce in term 1.
I20260927 18:44:48.133450 1363 leader_election.cc:302] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:48.133846 1736 raft_consensus.cc:2781] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:48.134279 1736 raft_consensus.cc:686] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 LEADER]: Becoming Leader. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Running, Role: LEADER
I20260927 18:44:48.134574 1736 consensus_queue.cc:237] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.137392 1133 catalog_manager.cc:5495] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194). New cstate: current_term: 1 leader_uuid: "9df85f6ae6714424af21b4b9b5d188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260927 18:44:48.184857 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:48.179674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.31.26.194:35039)
0927 18:44:48.180268 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:48.180272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:48.180291 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:44:48.180343 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:48.180346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:48.180409 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:48.180492 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:48.180817 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:48.181590 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:48.182827 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:48.183269 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:48.183361 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:48.183773 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:48.183879 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:48.183946 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:48.184093 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 18:44:48.184153 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":375,"threads_started":1}
I20260927 18:44:48.194242 1573 raft_consensus.cc:1261] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Refusing update from remote peer 9df85f6ae6714424af21b4b9b5d188ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:48.194334 1291 raft_consensus.cc:1261] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Refusing update from remote peer 9df85f6ae6714424af21b4b9b5d188ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:48.194805 1736 consensus_queue.cc:1035] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:48.195067 1737 consensus_queue.cc:1035] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:48.243886 1134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:48.244092 1134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:48.246811 1134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Sending DeleteTablet for 3 replicas of tablet 557bda0fc43a46268d025443064e2a49
I20260927 18:44:48.247684 1411 tablet_service.cc:1514] Processing DeleteTablet for tablet 557bda0fc43a46268d025443064e2a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:48 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:42834
I20260927 18:44:48.247727 1272 tablet_service.cc:1514] Processing DeleteTablet for tablet 557bda0fc43a46268d025443064e2a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:48 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:56138
I20260927 18:44:48.247766 1549 tablet_service.cc:1514] Processing DeleteTablet for tablet 557bda0fc43a46268d025443064e2a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:48 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:43436
I20260927 18:44:48.247985 1719 tablet_replica.cc:330] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: stopping tablet replica
I20260927 18:44:48.248080 1720 tablet_replica.cc:330] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: stopping tablet replica
I20260927 18:44:48.248086 1718 tablet_replica.cc:330] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: stopping tablet replica
I20260927 18:44:48.248194 1719 raft_consensus.cc:2229] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:48.248279 1720 raft_consensus.cc:2229] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:48.248302 1718 raft_consensus.cc:2229] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:48.248376 1719 raft_consensus.cc:2258] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:48.248443 1720 raft_consensus.cc:2258] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:48.248631 1718 raft_consensus.cc:2258] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:48.248662 1134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:44:48.249367 1134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:44:48.249606 1719 ts_tablet_manager.cc:1905] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:48.249814 1720 ts_tablet_manager.cc:1905] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:48.249820 1718 ts_tablet_manager.cc:1905] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:48.256515 1720 ts_tablet_manager.cc:1918] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:48.256521 1719 ts_tablet_manager.cc:1918] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:48.256687 1719 log.cc:1196] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/557bda0fc43a46268d025443064e2a49
I20260927 18:44:48.256687 1720 log.cc:1196] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/557bda0fc43a46268d025443064e2a49
I20260927 18:44:48.256695 1718 ts_tablet_manager.cc:1918] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:44:48.256857 1718 log.cc:1196] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/557bda0fc43a46268d025443064e2a49
I20260927 18:44:48.257186 1719 ts_tablet_manager.cc:1939] T 557bda0fc43a46268d025443064e2a49 P 905cda65bc8042dbb8967f5032022eaf: Deleting consensus metadata
I20260927 18:44:48.257305 1718 ts_tablet_manager.cc:1939] T 557bda0fc43a46268d025443064e2a49 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting consensus metadata
I20260927 18:44:48.257438 1720 ts_tablet_manager.cc:1939] T 557bda0fc43a46268d025443064e2a49 P 4db34bc436044cf3a5a7e4f49e595744: Deleting consensus metadata
I20260927 18:44:48.258035 1272 tablet_service.cc:1467] Processing CreateTablet for tablet db3688aeadab42ada01cd43f89d12091 (DEFAULT_TABLE table=test-table [id=609d1627489149ea84dde90094109f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:48.258087 1411 tablet_service.cc:1467] Processing CreateTablet for tablet db3688aeadab42ada01cd43f89d12091 (DEFAULT_TABLE table=test-table [id=609d1627489149ea84dde90094109f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:48.258507 1411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3688aeadab42ada01cd43f89d12091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:48.258508 1272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3688aeadab42ada01cd43f89d12091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:48.258909 1120 catalog_manager.cc:4841] TS 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701): tablet 557bda0fc43a46268d025443064e2a49 (table test-table [id=64aa1e701cbc42b6bd2c26ae8f8626ae]) successfully deleted
I20260927 18:44:48.259398 1549 tablet_service.cc:1467] Processing CreateTablet for tablet db3688aeadab42ada01cd43f89d12091 (DEFAULT_TABLE table=test-table [id=609d1627489149ea84dde90094109f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:44:48.259521 1119 catalog_manager.cc:4841] TS 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039): tablet 557bda0fc43a46268d025443064e2a49 (table test-table [id=64aa1e701cbc42b6bd2c26ae8f8626ae]) successfully deleted
I20260927 18:44:48.259887 1549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3688aeadab42ada01cd43f89d12091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:48.262108 1121 catalog_manager.cc:4841] TS 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917): tablet 557bda0fc43a46268d025443064e2a49 (table test-table [id=64aa1e701cbc42b6bd2c26ae8f8626ae]) successfully deleted
I20260927 18:44:48.264361 1731 tablet_bootstrap.cc:492] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Bootstrap starting.
I20260927 18:44:48.265605 1729 tablet_bootstrap.cc:492] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Bootstrap starting.
I20260927 18:44:48.265976 1731 tablet_bootstrap.cc:654] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:48.266583 1730 tablet_bootstrap.cc:492] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Bootstrap starting.
I20260927 18:44:48.267153 1729 tablet_bootstrap.cc:654] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:48.268334 1730 tablet_bootstrap.cc:654] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:48.269461 1731 tablet_bootstrap.cc:492] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: No bootstrap required, opened a new log
I20260927 18:44:48.269634 1731 ts_tablet_manager.cc:1397] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:48.270484 1731 raft_consensus.cc:348] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.270730 1731 raft_consensus.cc:374] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:48.270836 1731 raft_consensus.cc:729] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df85f6ae6714424af21b4b9b5d188ce, State: Initialized, Role: FOLLOWER
I20260927 18:44:48.271049 1730 tablet_bootstrap.cc:492] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: No bootstrap required, opened a new log
I20260927 18:44:48.271224 1730 ts_tablet_manager.cc:1397] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:48.271127 1731 consensus_queue.cc:260] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.271332 1729 tablet_bootstrap.cc:492] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: No bootstrap required, opened a new log
I20260927 18:44:48.271639 1729 ts_tablet_manager.cc:1397] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 18:44:48.271675 1731 ts_tablet_manager.cc:1428] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 18:44:48.272038 1730 raft_consensus.cc:348] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.272275 1730 raft_consensus.cc:374] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:48.272368 1730 raft_consensus.cc:729] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Initialized, Role: FOLLOWER
I20260927 18:44:48.272357 1729 raft_consensus.cc:348] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.272584 1729 raft_consensus.cc:374] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:48.272662 1729 raft_consensus.cc:729] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905cda65bc8042dbb8967f5032022eaf, State: Initialized, Role: FOLLOWER
I20260927 18:44:48.272615 1730 consensus_queue.cc:260] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.272914 1729 consensus_queue.cc:260] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.273154 1730 ts_tablet_manager.cc:1428] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:44:48.274518 1729 ts_tablet_manager.cc:1428] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:44:48.317509 1713 raft_consensus.cc:482] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:44:48.317739 1713 raft_consensus.cc:504] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.318409 1713 leader_election.cc:288] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:48.318926 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3688aeadab42ada01cd43f89d12091" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf" is_pre_election: true
I20260927 18:44:48.318926 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3688aeadab42ada01cd43f89d12091" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df85f6ae6714424af21b4b9b5d188ce" is_pre_election: true
I20260927 18:44:48.319207 1436 raft_consensus.cc:2454] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 0.
I20260927 18:44:48.319207 1291 raft_consensus.cc:2454] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 0.
I20260927 18:44:48.319649 1501 leader_election.cc:302] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:48.319944 1713 raft_consensus.cc:2781] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:44:48.320067 1713 raft_consensus.cc:482] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:44:48.320178 1713 raft_consensus.cc:3037] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:48.322224 1713 raft_consensus.cc:504] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.322760 1713 leader_election.cc:288] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 election: Requested vote from peers 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917), 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039)
I20260927 18:44:48.323271 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3688aeadab42ada01cd43f89d12091" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905cda65bc8042dbb8967f5032022eaf"
I20260927 18:44:48.323267 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3688aeadab42ada01cd43f89d12091" candidate_uuid: "4db34bc436044cf3a5a7e4f49e595744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df85f6ae6714424af21b4b9b5d188ce"
I20260927 18:44:48.323499 1291 raft_consensus.cc:3037] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:48.323499 1436 raft_consensus.cc:3037] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:48.326526 1291 raft_consensus.cc:2454] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 1.
I20260927 18:44:48.326524 1436 raft_consensus.cc:2454] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db34bc436044cf3a5a7e4f49e595744 in term 1.
I20260927 18:44:48.327045 1501 leader_election.cc:302] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db34bc436044cf3a5a7e4f49e595744, 9df85f6ae6714424af21b4b9b5d188ce; no voters:
I20260927 18:44:48.327461 1713 raft_consensus.cc:2781] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:48.327637 1713 raft_consensus.cc:686] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 LEADER]: Becoming Leader. State: Replica: 4db34bc436044cf3a5a7e4f49e595744, State: Running, Role: LEADER
I20260927 18:44:48.327903 1713 consensus_queue.cc:237] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } }
I20260927 18:44:48.330478 1133 catalog_manager.cc:5495] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195). New cstate: current_term: 1 leader_uuid: "4db34bc436044cf3a5a7e4f49e595744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4db34bc436044cf3a5a7e4f49e595744" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260927 18:44:48.362777 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:48.358109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.31.26.195:44701)
0927 18:44:48.358654 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:48.358658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:48.358676 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:44:48.358723 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:48.358726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:48.358784 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:48.358865 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:48.359177 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:48.359958 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:48.361208 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:48.361597 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:48.361691 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:48.362138 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:48.362246 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:48.362327 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:48.362452 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 18:44:48.362513 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
I20260927 18:44:48.372470 1291 raft_consensus.cc:1261] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Refusing update from remote peer 4db34bc436044cf3a5a7e4f49e595744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:48.372545 1436 raft_consensus.cc:1261] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Refusing update from remote peer 4db34bc436044cf3a5a7e4f49e595744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:48.373020 1713 consensus_queue.cc:1035] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905cda65bc8042dbb8967f5032022eaf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:48.373345 1701 consensus_queue.cc:1035] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df85f6ae6714424af21b4b9b5d188ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:44:48.393356 1699 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:44:49.977278 1133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:49.977496 1133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:44:49.981710 1133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fca1ea600e24b048067b8dac3f30ec3: Sending DeleteTablet for 3 replicas of tablet db3688aeadab42ada01cd43f89d12091
I20260927 18:44:49.982677 1411 tablet_service.cc:1514] Processing DeleteTablet for tablet db3688aeadab42ada01cd43f89d12091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:49 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:42834
I20260927 18:44:49.982677 1272 tablet_service.cc:1514] Processing DeleteTablet for tablet db3688aeadab42ada01cd43f89d12091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:49 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:56138
I20260927 18:44:49.982837 1549 tablet_service.cc:1514] Processing DeleteTablet for tablet db3688aeadab42ada01cd43f89d12091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:44:49 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:43436
I20260927 18:44:49.983495 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1206
I20260927 18:44:49.996840 1772 tablet_replica.cc:330] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: stopping tablet replica
I20260927 18:44:49.997143 1772 raft_consensus.cc:2229] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:49.997228 1773 tablet_replica.cc:330] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: stopping tablet replica
I20260927 18:44:49.997605 1772 raft_consensus.cc:2258] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:49.997617 1773 raft_consensus.cc:2229] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:44:49.997954 1773 raft_consensus.cc:2258] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:49.999037 1773 ts_tablet_manager.cc:1905] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:49.999084 1772 ts_tablet_manager.cc:1905] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:50.006783 1772 ts_tablet_manager.cc:1918] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:44:50.006783 1773 ts_tablet_manager.cc:1918] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:44:50.006969 1772 log.cc:1196] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/db3688aeadab42ada01cd43f89d12091
I20260927 18:44:50.007153 1773 log.cc:1196] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/db3688aeadab42ada01cd43f89d12091
I20260927 18:44:50.007499 1772 ts_tablet_manager.cc:1939] T db3688aeadab42ada01cd43f89d12091 P 9df85f6ae6714424af21b4b9b5d188ce: Deleting consensus metadata
I20260927 18:44:50.007720 1773 ts_tablet_manager.cc:1939] T db3688aeadab42ada01cd43f89d12091 P 905cda65bc8042dbb8967f5032022eaf: Deleting consensus metadata
I20260927 18:44:50.008126 1774 tablet_replica.cc:330] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: stopping tablet replica
I20260927 18:44:50.008338 1774 raft_consensus.cc:2229] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:44:50.008803 1774 raft_consensus.cc:2258] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:44:50.008750 1119 catalog_manager.cc:4841] TS 9df85f6ae6714424af21b4b9b5d188ce (127.31.26.194:35039): tablet db3688aeadab42ada01cd43f89d12091 (table test-table [id=609d1627489149ea84dde90094109f5b]) successfully deleted
I20260927 18:44:50.008829 1121 catalog_manager.cc:4841] TS 905cda65bc8042dbb8967f5032022eaf (127.31.26.193:45917): tablet db3688aeadab42ada01cd43f89d12091 (table test-table [id=609d1627489149ea84dde90094109f5b]) successfully deleted
I20260927 18:44:50.009716 1774 ts_tablet_manager.cc:1905] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:44:50.014580 1774 ts_tablet_manager.cc:1918] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:44:50.014739 1774 log.cc:1196] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/db3688aeadab42ada01cd43f89d12091
I20260927 18:44:50.015231 1774 ts_tablet_manager.cc:1939] T db3688aeadab42ada01cd43f89d12091 P 4db34bc436044cf3a5a7e4f49e595744: Deleting consensus metadata
I20260927 18:44:50.016244 1120 catalog_manager.cc:4841] TS 4db34bc436044cf3a5a7e4f49e595744 (127.31.26.195:44701): tablet db3688aeadab42ada01cd43f89d12091 (table test-table [id=609d1627489149ea84dde90094109f5b]) successfully deleted
Sep 27 18:44:50 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:50.025173 1775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:49.984661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.31.26.193:45917)
0927 18:44:49.997008 (+ 12347us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:49.997018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:49.997063 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:44:49.997199 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:49.997207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:49.997383 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:49.997552 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:50.000245 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:50.002037 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:50.004902 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:50.006123 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:50.006295 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:50.018533 (+ 12238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:50.018578 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:50.018597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:50.018680 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:50.022144 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:50.023004 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:50.023013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:50.023015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:50.023181 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:50.023609 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:50.023613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:50.023615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:50.023987 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:50.024197 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:50.024501 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:44:50.024678 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12033,"thread_start_us":11896,"threads_started":1}
W20260927 18:44:50.025888 1337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56152
W20260927 18:44:50.026691 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:50.026947 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 1206
I20260927 18:44:50.046362 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1346
Sep 27 18:44:50 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:44:50.063015 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:50.047235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.31.26.194:35039)
0927 18:44:50.047753 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:50.047758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:50.047776 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:44:50.047830 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:50.047833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:50.047892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:50.048030 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:50.050155 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:50.050906 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:50.052173 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:50.052612 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:50.052767 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:50.057212 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:50.057229 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:50.057246 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:50.057283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:50.060037 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:50.060751 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:50.060764 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:50.060773 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:50.060912 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:50.061357 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:50.061365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:50.061368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:50.061746 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:50.062075 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:50.062393 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 18:44:50.062623 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":239,"threads_started":1}
W20260927 18:44:50.063735 1476 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42862
W20260927 18:44:50.064455 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:50.064630 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 1346
I20260927 18:44:50.087399 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1485
Sep 27 18:44:50 dist-test-slave-rfd5 krb5kdc[1090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534687, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:44:50.103623 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:50.088438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.31.26.195:44701)
0927 18:44:50.088928 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:50.088933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:50.088949 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:50.089000 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:50.089004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:50.089063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:50.089220 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:50.090583 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:50.091404 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:50.092710 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:50.093168 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:50.093291 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:50.097761 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:50.097775 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:50.097791 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:50.097832 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:50.100719 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:50.101540 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:50.101548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:50.101550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:50.101698 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:50.102079 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:50.102086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:50.102089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:50.102442 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:50.102762 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:50.103054 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 18:44:50.103267 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
W20260927 18:44:50.104393 1615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43460
W20260927 18:44:50.105008 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:50.105226 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 1485
I20260927 18:44:50.134085 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1103
I20260927 18:44:50.135372 1175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:50.250327 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 1103
2026-09-27T18:44:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8285 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:50 dist-test-slave-rfd5 krb5kdc[1782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:50 dist-test-slave-rfd5 krb5kdc[1782](info): set up 2 sockets
Sep 27 18:44:50 dist-test-slave-rfd5 krb5kdc[1782](info): commencing operation
krb5kdc: starting...
W20260927 18:44:51.341549 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:51 dist-test-slave-rfd5 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:51.537295 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44325
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:51.865435 1795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:51.865808 1795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:51.865974 1795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:51.873313 1795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:51.873494 1795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:51.873608 1795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:51.873714 1795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:51.873801 1795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:51.883912 1795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:51.885444 1795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:51.886791 1795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:51.899184 1803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:51.899268 1801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:51.899457 1800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:51.899933 1795 server_base.cc:1034] running on GCE node
I20260927 18:44:51.900775 1795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:51.902304 1795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:51.903586 1795 hybrid_clock.cc:648] HybridClock initialized: now 1790534691903537 us; error 57 us; skew 500 ppm
I20260927 18:44:51.904018 1795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:51 dist-test-slave-rfd5 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534691, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:51.907434 1795 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:51.908968 1795 webserver.cc:460] Webserver started at http://127.31.26.254:46603/ using document root <none> and password file <none>
I20260927 18:44:51.909718 1795 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:51.909816 1795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:51.910076 1795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:51.913031 1795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "78055af31369475f8699417d09a5ecd8"
format_stamp: "Formatted at 2026-09-27 18:44:51 on dist-test-slave-rfd5"
server_key: "3c1c0b6f54560ebd2a6deea5be234196"
server_key_iv: "a5b1119eaa71d7a1cbe684de1abd9e3e"
server_key_version: "encryptionkey@0"
I20260927 18:44:51.913828 1795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "78055af31369475f8699417d09a5ecd8"
format_stamp: "Formatted at 2026-09-27 18:44:51 on dist-test-slave-rfd5"
server_key: "3c1c0b6f54560ebd2a6deea5be234196"
server_key_iv: "a5b1119eaa71d7a1cbe684de1abd9e3e"
server_key_version: "encryptionkey@0"
I20260927 18:44:51.919050 1795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:44:51.922566 1810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:51.924010 1795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:51.924160 1795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "78055af31369475f8699417d09a5ecd8"
format_stamp: "Formatted at 2026-09-27 18:44:51 on dist-test-slave-rfd5"
server_key: "3c1c0b6f54560ebd2a6deea5be234196"
server_key_iv: "a5b1119eaa71d7a1cbe684de1abd9e3e"
server_key_version: "encryptionkey@0"
I20260927 18:44:51.924346 1795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:51.955278 1795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:51.959720 1795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:51.959986 1795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:51.978950 1795 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44325
I20260927 18:44:51.978955 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44325 every 8 connection(s)
I20260927 18:44:51.980362 1795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:51.984158 1862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:51.986701 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 1795
I20260927 18:44:51.986934 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:51.987748 31851 external_mini_cluster.cc:1428] Setting key 163621457e7c24970047c48f94096bbc
I20260927 18:44:51.997625 1862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8: Bootstrap starting.
I20260927 18:44:52.001582 1862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:52.002909 1862 log.cc:824] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:44:51 dist-test-slave-rfd5 krb5kdc[1782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:52.006680 1862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8: No bootstrap required, opened a new log
I20260927 18:44:52.013816 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:51.990005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.31.26.254:44325)
0927 18:44:51.990858 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:51.990871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:51.990918 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:44:51.992654 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:51.992679 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:51.993212 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:51.993596 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:51.995452 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:51.996784 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:51.998573 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:51.999287 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:51.999429 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:52.004606 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:52.004616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:52.004636 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:52.004679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:52.008834 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:52.009710 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:52.009719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:52.009728 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:52.009952 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:52.010332 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:52.010337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:52.010341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:52.011156 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:52.011502 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:52.011791 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:44:52.012265 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":394,"threads_started":1}
I20260927 18:44:52.017529 1862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78055af31369475f8699417d09a5ecd8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44325 } }
I20260927 18:44:52.017894 1862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:52.017980 1862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78055af31369475f8699417d09a5ecd8, State: Initialized, Role: FOLLOWER
I20260927 18:44:52.018484 1862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78055af31369475f8699417d09a5ecd8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44325 } }
I20260927 18:44:52.018690 1862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:52.018780 1862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:52.018906 1862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:52.021159 1862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78055af31369475f8699417d09a5ecd8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44325 } }
I20260927 18:44:52.021857 1862 leader_election.cc:302] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78055af31369475f8699417d09a5ecd8; no voters:
I20260927 18:44:52.022519 1862 leader_election.cc:288] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:52.022670 1867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:52.023470 1867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [term 1 LEADER]: Becoming Leader. State: Replica: 78055af31369475f8699417d09a5ecd8, State: Running, Role: LEADER
I20260927 18:44:52.024010 1867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78055af31369475f8699417d09a5ecd8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44325 } }
I20260927 18:44:52.024730 1862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:52.026970 1869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78055af31369475f8699417d09a5ecd8. Latest consensus state: current_term: 1 leader_uuid: "78055af31369475f8699417d09a5ecd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78055af31369475f8699417d09a5ecd8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44325 } } }
I20260927 18:44:52.027484 1869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:52.027381 1868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78055af31369475f8699417d09a5ecd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78055af31369475f8699417d09a5ecd8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44325 } } }
I20260927 18:44:52.027664 1868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:52.030709 1875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:52.037951 1875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:52.049393 1875 catalog_manager.cc:1347] Generated new cluster ID: 44fb68a84f154afba98285cd2260b63a
I20260927 18:44:52.049540 1875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:52.063040 1875 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:52.064430 1875 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:52.073073 1875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78055af31369475f8699417d09a5ecd8: Generated new TSK 0
I20260927 18:44:52.074036 1875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:44:52.121218 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44325
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:52.457242 1888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:52.457556 1888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:52.457723 1888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:52.465040 1888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:52.465186 1888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:52.465394 1888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:44:52.475251 1888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:52.476986 1888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:52.478394 1888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:52.491600 1893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:52.491688 1894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:52.491660 1896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:52.492771 1888 server_base.cc:1034] running on GCE node
I20260927 18:44:52.493521 1888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:52.494644 1888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:52.495842 1888 hybrid_clock.cc:648] HybridClock initialized: now 1790534692495774 us; error 59 us; skew 500 ppm
I20260927 18:44:52.496170 1888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:52 dist-test-slave-rfd5 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534692, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:52.499226 1888 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:44:52.500764 1888 webserver.cc:460] Webserver started at http://127.31.26.193:46401/ using document root <none> and password file <none>
I20260927 18:44:52.501541 1888 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:52.501686 1888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:52.501995 1888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:52.504932 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "c7103e9c426d40b685362d68f0eaba55"
format_stamp: "Formatted at 2026-09-27 18:44:52 on dist-test-slave-rfd5"
server_key: "eef6d5d26491e9edc4ad57e8e8005fd5"
server_key_iv: "89a062978063e0fc2ce1715b540a8d28"
server_key_version: "encryptionkey@0"
I20260927 18:44:52.505817 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "c7103e9c426d40b685362d68f0eaba55"
format_stamp: "Formatted at 2026-09-27 18:44:52 on dist-test-slave-rfd5"
server_key: "eef6d5d26491e9edc4ad57e8e8005fd5"
server_key_iv: "89a062978063e0fc2ce1715b540a8d28"
server_key_version: "encryptionkey@0"
I20260927 18:44:52.511314 1888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:44:52.515515 1903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:52.517263 1888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:52.517433 1888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "c7103e9c426d40b685362d68f0eaba55"
format_stamp: "Formatted at 2026-09-27 18:44:52 on dist-test-slave-rfd5"
server_key: "eef6d5d26491e9edc4ad57e8e8005fd5"
server_key_iv: "89a062978063e0fc2ce1715b540a8d28"
server_key_version: "encryptionkey@0"
I20260927 18:44:52.517628 1888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:52.531175 1888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:52.535881 1888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:52.536146 1888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:52.537159 1888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:52.538492 1888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:52.538568 1888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:52.538658 1888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:52.538702 1888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:52.571511 1888 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:43087
I20260927 18:44:52.571516 2015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:43087 every 8 connection(s)
I20260927 18:44:52.572999 1888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:52.581971 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 1888
I20260927 18:44:52.582222 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:52.583007 31851 external_mini_cluster.cc:1428] Setting key c4dcfff84ebbc3c7ee877dc2c22a75ff
Sep 27 18:44:52 dist-test-slave-rfd5 krb5kdc[1782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534692, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:52.595052 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:52.575757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:57579 (local address 127.31.26.254:44325)
0927 18:44:52.576124 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:52.576129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:52.577333 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 18:44:52.579173 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:52.579178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:52.579256 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:52.579357 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:52.580910 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:52.581717 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:52.584076 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:52.584533 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:52.584645 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:52.588414 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:52.588426 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:52.588434 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:52.588489 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:52.591506 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:52.592336 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:52.592345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:52.592352 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:52.592507 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:52.592947 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:52.592954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:52.592961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:52.593375 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:52.593538 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:52.594314 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 18:44:52.594584 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":185,"threads_started":1}
I20260927 18:44:52.595758 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:52.576399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44325 (local address 127.31.26.193:57579)
0927 18:44:52.577155 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:52.577215 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:52.578871 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:52.579595 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:52.579604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:52.580134 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:52.580745 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:52.580764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:52.581899 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:52.581906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:52.583923 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:52.583941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:52.584785 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:52.584794 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:52.585033 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:52.585846 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:52.585871 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:52.588127 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:52.591662 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:52.591670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:52.591686 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:52.592174 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:52.592641 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:52.592644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:52.592646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:52.592834 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:52.593518 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:52.593525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:52.593886 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 18:44:52.594153 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 18:44:52.594575 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":263,"threads_started":1}
I20260927 18:44:52.597404 2016 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44325
I20260927 18:44:52.597765 2016 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:52.598699 2016 heartbeater.cc:511] Master 127.31.26.254:44325 requested a full tablet report, sending...
I20260927 18:44:52.601052 1827 ts_manager.cc:194] Registered new tserver with Master: c7103e9c426d40b685362d68f0eaba55 (127.31.26.193:43087)
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:44:52.602644 1827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:57579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:44:52.628533 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44325
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:52.958532 2022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:52.958900 2022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:52.959084 2022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:52.966373 2022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:52.966560 2022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:52.966746 2022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:44:52.976637 2022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:52.978029 2022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:52.979450 2022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:52.992764 2028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:52.993197 2027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:52.993357 2030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:52.994099 2022 server_base.cc:1034] running on GCE node
I20260927 18:44:52.994783 2022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:52.995816 2022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:52.997016 2022 hybrid_clock.cc:648] HybridClock initialized: now 1790534692996946 us; error 61 us; skew 500 ppm
I20260927 18:44:52.997400 2022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:52 dist-test-slave-rfd5 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534692, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:53.000540 2022 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:44:53.002112 2022 webserver.cc:460] Webserver started at http://127.31.26.194:34539/ using document root <none> and password file <none>
I20260927 18:44:53.002852 2022 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:53.002998 2022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:53.003311 2022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:53.006362 2022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "61535fff1c7945ce843f5de106d75b05"
format_stamp: "Formatted at 2026-09-27 18:44:52 on dist-test-slave-rfd5"
server_key: "57774387febd9d2cea5d36ea6fcddd39"
server_key_iv: "ed2ccaf6d2fce6a69d334347d84952b1"
server_key_version: "encryptionkey@0"
I20260927 18:44:53.007218 2022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "61535fff1c7945ce843f5de106d75b05"
format_stamp: "Formatted at 2026-09-27 18:44:52 on dist-test-slave-rfd5"
server_key: "57774387febd9d2cea5d36ea6fcddd39"
server_key_iv: "ed2ccaf6d2fce6a69d334347d84952b1"
server_key_version: "encryptionkey@0"
I20260927 18:44:53.012583 2022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:44:53.016240 2037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:53.017688 2022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:53.017874 2022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "61535fff1c7945ce843f5de106d75b05"
format_stamp: "Formatted at 2026-09-27 18:44:52 on dist-test-slave-rfd5"
server_key: "57774387febd9d2cea5d36ea6fcddd39"
server_key_iv: "ed2ccaf6d2fce6a69d334347d84952b1"
server_key_version: "encryptionkey@0"
I20260927 18:44:53.018075 2022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:53.031440 2022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:53.036149 2022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:53.036407 2022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:53.037415 2022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:53.038725 2022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:53.038800 2022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:53.038893 2022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:53.038937 2022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:53.072837 2022 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:35559
I20260927 18:44:53.072878 2149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:35559 every 8 connection(s)
I20260927 18:44:53.074255 2022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:53.078413 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2022
I20260927 18:44:53.078626 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:53.079103 31851 external_mini_cluster.cc:1428] Setting key 7d5d69add497b706c0771cc045e7f713
Sep 27 18:44:53 dist-test-slave-rfd5 krb5kdc[1782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534692, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:53.094653 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.076922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:60671 (local address 127.31.26.254:44325)
0927 18:44:53.077205 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:53.077210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:53.078397 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 18:44:53.080471 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:53.080475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:53.080547 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:53.080631 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:53.082279 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.082972 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.084544 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.084978 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.085136 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:53.088731 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:53.088736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:53.088740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:53.088771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:53.091432 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:53.092219 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:53.092225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:53.092227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:53.092360 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:53.092820 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:53.092832 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:53.092835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:53.093176 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:53.093331 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:53.094072 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 18:44:53.094307 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:44:53.095854 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.077502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44325 (local address 127.31.26.194:60671)
0927 18:44:53.078235 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:53.078281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:53.080177 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:53.080908 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:53.080917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:53.081522 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:53.082103 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:53.082120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.083134 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.083139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:53.084390 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:53.084402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.085235 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.085242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:53.085482 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:53.086193 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:53.086216 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:53.088497 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:53.091594 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:53.091604 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:53.091624 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:53.092077 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:53.092492 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:53.092497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:53.092501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:53.092699 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:53.093305 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:53.093314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:53.093685 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 18:44:53.093959 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 18:44:53.094553 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":287,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:44:53.097707 2150 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44325
I20260927 18:44:53.098075 2150 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:53.098837 2150 heartbeater.cc:511] Master 127.31.26.254:44325 requested a full tablet report, sending...
I20260927 18:44:53.100419 1827 ts_manager.cc:194] Registered new tserver with Master: 61535fff1c7945ce843f5de106d75b05 (127.31.26.194:35559)
I20260927 18:44:53.101301 1827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:60671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:44:53.122983 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44325
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:53.457695 2155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:53.457994 2155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:53.458091 2155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:53.465413 2155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:53.465551 2155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:53.465670 2155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:44:53.475822 2155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:53.477185 2155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:53.478523 2155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:53.492642 2160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:53.492544 2161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:53.492575 2163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:53.494071 2155 server_base.cc:1034] running on GCE node
I20260927 18:44:53.494812 2155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:53.495942 2155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:53.497219 2155 hybrid_clock.cc:648] HybridClock initialized: now 1790534693497146 us; error 51 us; skew 500 ppm
I20260927 18:44:53.497602 2155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:53 dist-test-slave-rfd5 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534693, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:53.500850 2155 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:44:53.502686 2155 webserver.cc:460] Webserver started at http://127.31.26.195:43217/ using document root <none> and password file <none>
I20260927 18:44:53.503446 2155 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:53.503605 2155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:53.503931 2155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:53.507085 2155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "cc549c745650457f9b66647261459e3f"
format_stamp: "Formatted at 2026-09-27 18:44:53 on dist-test-slave-rfd5"
server_key: "e0c3fb45785c37c414ffaabc78cffa6d"
server_key_iv: "68fa1d9aa69ec2b430d38ac8bf6e2fcc"
server_key_version: "encryptionkey@0"
I20260927 18:44:53.507977 2155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "cc549c745650457f9b66647261459e3f"
format_stamp: "Formatted at 2026-09-27 18:44:53 on dist-test-slave-rfd5"
server_key: "e0c3fb45785c37c414ffaabc78cffa6d"
server_key_iv: "68fa1d9aa69ec2b430d38ac8bf6e2fcc"
server_key_version: "encryptionkey@0"
I20260927 18:44:53.513615 2155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:44:53.517470 2170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:53.519037 2155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:53.519253 2155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "cc549c745650457f9b66647261459e3f"
format_stamp: "Formatted at 2026-09-27 18:44:53 on dist-test-slave-rfd5"
server_key: "e0c3fb45785c37c414ffaabc78cffa6d"
server_key_iv: "68fa1d9aa69ec2b430d38ac8bf6e2fcc"
server_key_version: "encryptionkey@0"
I20260927 18:44:53.519519 2155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:53.537294 2155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:53.541986 2155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:53.542307 2155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:53.543313 2155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:53.544718 2155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:53.544831 2155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:53.544987 2155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:53.545063 2155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:53.581068 2155 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:46493
I20260927 18:44:53.581172 2282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:46493 every 8 connection(s)
I20260927 18:44:53.582546 2155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:53.586918 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2155
I20260927 18:44:53.587127 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:53.587703 31851 external_mini_cluster.cc:1428] Setting key cae9d16f52761dee3ed5809652e5d047
Sep 27 18:44:53 dist-test-slave-rfd5 krb5kdc[1782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534693, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:53.604151 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.585323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:43355 (local address 127.31.26.254:44325)
0927 18:44:53.585532 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:53.585540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:53.586930 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 18:44:53.589339 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:53.589343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:53.589440 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:53.589541 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:53.591478 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.592245 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.593916 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.594318 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.594430 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:53.597975 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:53.597981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:53.597984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:53.598019 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:53.600686 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:53.601687 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:53.601691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:53.601693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:53.601845 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:53.602307 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:53.602318 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:53.602321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:53.602696 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:53.602855 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:53.603572 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 18:44:53.603782 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:44:53.605249 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:53.585894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44325 (local address 127.31.26.195:43355)
0927 18:44:53.586710 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:53.586777 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:53.588987 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:53.589794 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:53.589803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:53.590590 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:53.591300 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:53.591319 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.592391 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.592395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:53.593772 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:53.593785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:53.594525 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:53.594530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:53.594735 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:53.595398 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:53.595421 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:53.597725 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:53.600858 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:53.600866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:53.600882 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:53.601535 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:53.601974 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:53.601980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:53.601984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:53.602187 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:53.602835 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:53.602843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:53.603215 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 18:44:53.603497 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 18:44:53.604066 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260927 18:44:53.606037 2016 heartbeater.cc:503] Master 127.31.26.254:44325 was elected leader, sending a full tablet report...
I20260927 18:44:53.606776 2283 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44325
I20260927 18:44:53.607129 2283 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:53.607991 2283 heartbeater.cc:511] Master 127.31.26.254:44325 requested a full tablet report, sending...
I20260927 18:44:53.609385 1827 ts_manager.cc:194] Registered new tserver with Master: cc549c745650457f9b66647261459e3f (127.31.26.195:46493)
I20260927 18:44:53.610157 1827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:43355
I20260927 18:44:53.615899 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:44:53.648542 2018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:53.645753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.31.26.254:44325)
0927 18:44:53.646013 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:53.646022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:53.646417 (+ 395us) server_negotiation.cc:407] Connection header received
0927 18:44:53.646572 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:53.646576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:53.646699 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:53.646816 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:53.648254 (+ 1438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59086: BlockingRecv error: recv got EOF from 127.0.0.1:59086 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:44:53.651679 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1888
W20260927 18:44:53.654780 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1888: Not authorized: Client connection negotiation failed: client connection to 127.31.26.193:43087: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:44:53.655005 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 1888
I20260927 18:44:53.670321 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2022
W20260927 18:44:53.673116 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2022: Not authorized: Client connection negotiation failed: client connection to 127.31.26.194:35559: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:44:53.673300 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2022
I20260927 18:44:53.688477 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2155
W20260927 18:44:53.691262 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2155: Not authorized: Client connection negotiation failed: client connection to 127.31.26.195:46493: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:44:53.691426 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2155
I20260927 18:44:53.706597 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 1795
I20260927 18:44:53.707791 1857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:53.809247 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 1795
2026-09-27T18:44:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3554 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:53 dist-test-slave-rfd5 krb5kdc[2300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:53 dist-test-slave-rfd5 krb5kdc[2300](info): set up 2 sockets
Sep 27 18:44:53 dist-test-slave-rfd5 krb5kdc[2300](info): commencing operation
krb5kdc: starting...
W20260927 18:44:54.897442 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:55 dist-test-slave-rfd5 krb5kdc[2300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:55.089880 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35761
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:55.414705 2313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:55.415076 2313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:55.415248 2313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:55.422523 2313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:55.422698 2313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:55.422812 2313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:55.422917 2313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:55.422994 2313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:55.432828 2313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:55.434363 2313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:55.435792 2313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:55.447757 2318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:55.447824 2319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:55.447779 2321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:55.448658 2313 server_base.cc:1034] running on GCE node
I20260927 18:44:55.449626 2313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:55.451327 2313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:55.452529 2313 hybrid_clock.cc:648] HybridClock initialized: now 1790534695452488 us; error 36 us; skew 500 ppm
I20260927 18:44:55.452884 2313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:55 dist-test-slave-rfd5 krb5kdc[2300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534695, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:55.459218 2313 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:55.460765 2313 webserver.cc:460] Webserver started at http://127.31.26.254:40361/ using document root <none> and password file <none>
I20260927 18:44:55.461516 2313 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:55.461623 2313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:55.461880 2313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:55.464792 2313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "f72079f75c7d455d9d87a469144e4ff7"
format_stamp: "Formatted at 2026-09-27 18:44:55 on dist-test-slave-rfd5"
server_key: "6af7d78670fb31778067fbec76aea5f8"
server_key_iv: "a2c316213d038260d9421d0b3136ee9a"
server_key_version: "encryptionkey@0"
I20260927 18:44:55.465634 2313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "f72079f75c7d455d9d87a469144e4ff7"
format_stamp: "Formatted at 2026-09-27 18:44:55 on dist-test-slave-rfd5"
server_key: "6af7d78670fb31778067fbec76aea5f8"
server_key_iv: "a2c316213d038260d9421d0b3136ee9a"
server_key_version: "encryptionkey@0"
I20260927 18:44:55.470880 2313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:55.474388 2328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:55.475816 2313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:55.476009 2313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "f72079f75c7d455d9d87a469144e4ff7"
format_stamp: "Formatted at 2026-09-27 18:44:55 on dist-test-slave-rfd5"
server_key: "6af7d78670fb31778067fbec76aea5f8"
server_key_iv: "a2c316213d038260d9421d0b3136ee9a"
server_key_version: "encryptionkey@0"
I20260927 18:44:55.476226 2313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:55.500025 2313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:55.504172 2313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:55.504508 2313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:55.523958 2313 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:35761
I20260927 18:44:55.523995 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:35761 every 8 connection(s)
I20260927 18:44:55.525475 2313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:55.528779 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2313
I20260927 18:44:55.529033 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:55.529583 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:55.529738 31851 external_mini_cluster.cc:1428] Setting key 40ddfdac5ad11b5daa4dd1c65c848fd2
I20260927 18:44:55.544832 2380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7: Bootstrap starting.
Sep 27 18:44:55 dist-test-slave-rfd5 krb5kdc[2300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:55.549064 2380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:55.550478 2380 log.cc:824] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:55.554024 2380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7: No bootstrap required, opened a new log
I20260927 18:44:55.555156 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:55.531625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.31.26.254:35761)
0927 18:44:55.532509 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:55.532525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:55.532581 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:44:55.534739 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:55.534772 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:55.535343 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:55.535848 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:55.537308 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:55.539265 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:55.540628 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:55.541784 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:55.542005 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:55.546045 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:55.546057 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:55.546077 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:55.546125 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:55.550211 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:55.551234 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:55.551250 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:55.551257 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:55.551476 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:55.551931 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:55.551936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:55.551939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:55.552602 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:55.552887 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:55.553276 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:44:55.553711 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":368,"threads_started":1}
I20260927 18:44:55.559276 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:55.562646 2380 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72079f75c7d455d9d87a469144e4ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35761 } }
I20260927 18:44:55.563020 2380 raft_consensus.cc:374] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:55.563143 2380 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72079f75c7d455d9d87a469144e4ff7, State: Initialized, Role: FOLLOWER
I20260927 18:44:55.563726 2380 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72079f75c7d455d9d87a469144e4ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35761 } }
I20260927 18:44:55.563952 2380 raft_consensus.cc:388] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:55.564077 2380 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:55.564260 2380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:55.566516 2380 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72079f75c7d455d9d87a469144e4ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35761 } }
I20260927 18:44:55.567051 2380 leader_election.cc:302] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f72079f75c7d455d9d87a469144e4ff7; no voters:
I20260927 18:44:55.567715 2380 leader_election.cc:288] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:55.567830 2386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:55.568976 2386 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [term 1 LEADER]: Becoming Leader. State: Replica: f72079f75c7d455d9d87a469144e4ff7, State: Running, Role: LEADER
I20260927 18:44:55.569613 2386 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72079f75c7d455d9d87a469144e4ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35761 } }
I20260927 18:44:55.570175 2380 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:55.572492 2388 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72079f75c7d455d9d87a469144e4ff7. Latest consensus state: current_term: 1 leader_uuid: "f72079f75c7d455d9d87a469144e4ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72079f75c7d455d9d87a469144e4ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35761 } } }
I20260927 18:44:55.572919 2388 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:55.572703 2387 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f72079f75c7d455d9d87a469144e4ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72079f75c7d455d9d87a469144e4ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35761 } } }
I20260927 18:44:55.573118 2387 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:55.578399 2394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:55.584262 2394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:55.595345 2394 catalog_manager.cc:1347] Generated new cluster ID: c2328cc3654745feb8e05a168440df27
I20260927 18:44:55.595476 2394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:55.614069 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:55.601751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.31.26.254:35761)
0927 18:44:55.601923 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:55.601929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:55.605471 (+ 3542us) server_negotiation.cc:407] Connection header received
0927 18:44:55.605644 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:55.605650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:55.605756 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:55.605879 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:55.607195 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:55.608365 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:55.610182 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:55.610849 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:55.610996 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:55.612179 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:55.612192 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:55.612197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:55.613196 (+ 999us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:55.613258 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:55.613294 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:55.613413 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:55.613541 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 18:44:55.613647 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:44:55.648970 2394 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:55.650961 2394 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:55.663173 2394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f72079f75c7d455d9d87a469144e4ff7: Generated new TSK 0
I20260927 18:44:55.664414 2394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:55.681128 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2313
I20260927 18:44:55.682293 2375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:55.776389 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2313
2026-09-27T18:44:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1962 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:55 dist-test-slave-rfd5 krb5kdc[2414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:55 dist-test-slave-rfd5 krb5kdc[2414](info): set up 2 sockets
Sep 27 18:44:55 dist-test-slave-rfd5 krb5kdc[2414](info): commencing operation
krb5kdc: starting...
W20260927 18:44:56.857637 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:44:56 dist-test-slave-rfd5 krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:44:57.053164 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41605
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:57.382540 2427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:57.382908 2427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:57.383090 2427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:57.390429 2427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:57.390594 2427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:57.390686 2427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:57.390790 2427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:57.390857 2427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:57.400913 2427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:57.402365 2427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:57.403715 2427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:57.415938 2435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:57.416005 2433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:57.416059 2432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:57.416766 2427 server_base.cc:1034] running on GCE node
I20260927 18:44:57.417750 2427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:57.419265 2427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:57.420485 2427 hybrid_clock.cc:648] HybridClock initialized: now 1790534697420436 us; error 42 us; skew 500 ppm
I20260927 18:44:57.420840 2427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:57 dist-test-slave-rfd5 krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:57.424055 2427 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:44:57.425642 2427 webserver.cc:460] Webserver started at http://127.31.26.254:35537/ using document root <none> and password file <none>
I20260927 18:44:57.426294 2427 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:57.426403 2427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:57.426661 2427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:57.429620 2427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "838261cc01774425b0010bd504671c38"
format_stamp: "Formatted at 2026-09-27 18:44:57 on dist-test-slave-rfd5"
server_key: "caf9c35f6703a03bc0c98a2148418196"
server_key_iv: "c16c517a4a04737026e7a790090bfc29"
server_key_version: "encryptionkey@0"
I20260927 18:44:57.430315 2427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "838261cc01774425b0010bd504671c38"
format_stamp: "Formatted at 2026-09-27 18:44:57 on dist-test-slave-rfd5"
server_key: "caf9c35f6703a03bc0c98a2148418196"
server_key_iv: "c16c517a4a04737026e7a790090bfc29"
server_key_version: "encryptionkey@0"
I20260927 18:44:57.435952 2427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:44:57.439685 2442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:57.441236 2427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:44:57.441416 2427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "838261cc01774425b0010bd504671c38"
format_stamp: "Formatted at 2026-09-27 18:44:57 on dist-test-slave-rfd5"
server_key: "caf9c35f6703a03bc0c98a2148418196"
server_key_iv: "c16c517a4a04737026e7a790090bfc29"
server_key_version: "encryptionkey@0"
I20260927 18:44:57.441643 2427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:57.456318 2427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:57.460546 2427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:57.460853 2427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:57.479437 2427 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41605
I20260927 18:44:57.479450 2493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41605 every 8 connection(s)
I20260927 18:44:57.480863 2427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:57.482257 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2427
I20260927 18:44:57.482486 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:44:57.483311 31851 external_mini_cluster.cc:1428] Setting key e0d3e9754d298a11eae3a00b626babbc
I20260927 18:44:57.485996 2494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:44:57 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:57.499055 2494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38: Bootstrap starting.
I20260927 18:44:57.502584 2494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:57.503755 2494 log.cc:824] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:57.506939 2494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38: No bootstrap required, opened a new log
I20260927 18:44:57.507874 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:57.485521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.31.26.254:41605)
0927 18:44:57.486371 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:57.486387 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:57.486439 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:44:57.488741 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:57.488771 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:57.489366 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:57.489854 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:57.491218 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:57.493314 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:57.494578 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:57.495609 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:57.495821 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:57.500148 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:57.500156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:57.500171 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:57.500212 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:57.503430 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:57.504261 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:57.504268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:57.504275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:57.504462 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:57.504924 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:57.504930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:57.504933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:57.505541 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:57.505866 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:57.506293 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 18:44:57.506721 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":356,"threads_started":1}
I20260927 18:44:57.515662 2494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838261cc01774425b0010bd504671c38" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41605 } }
I20260927 18:44:57.516064 2494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:57.516184 2494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838261cc01774425b0010bd504671c38, State: Initialized, Role: FOLLOWER
I20260927 18:44:57.516738 2494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838261cc01774425b0010bd504671c38" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41605 } }
I20260927 18:44:57.516968 2494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:57.517138 2494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:57.517309 2494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:57.519536 2494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838261cc01774425b0010bd504671c38" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41605 } }
I20260927 18:44:57.520053 2494 leader_election.cc:302] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838261cc01774425b0010bd504671c38; no voters:
I20260927 18:44:57.520759 2494 leader_election.cc:288] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:57.520885 2499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:57.521637 2499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [term 1 LEADER]: Becoming Leader. State: Replica: 838261cc01774425b0010bd504671c38, State: Running, Role: LEADER
I20260927 18:44:57.522188 2499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838261cc01774425b0010bd504671c38" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41605 } }
I20260927 18:44:57.522971 2494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:57.524887 2500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838261cc01774425b0010bd504671c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838261cc01774425b0010bd504671c38" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41605 } } }
I20260927 18:44:57.525027 2501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838261cc01774425b0010bd504671c38. Latest consensus state: current_term: 1 leader_uuid: "838261cc01774425b0010bd504671c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838261cc01774425b0010bd504671c38" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41605 } } }
I20260927 18:44:57.525295 2500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:57.525310 2501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:57.528784 2507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:57.533910 2507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:57.542654 2507 catalog_manager.cc:1347] Generated new cluster ID: fa42860e75774f2590f63296d376e5c3
I20260927 18:44:57.542796 2507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:57.562853 2507 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:57.564247 2507 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:57.573064 2507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838261cc01774425b0010bd504671c38: Generated new TSK 0
I20260927 18:44:57.574064 2507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:44:57.626997 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41605
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:57.974280 2520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:57.974740 2520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:57.974934 2520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:57.987601 2520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:57.987766 2520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:57.988019 2520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:44:58.006482 2520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:58.008476 2520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:58.010461 2520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:58.025876 2525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:58.025782 2528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:58.025835 2526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:58.027407 2520 server_base.cc:1034] running on GCE node
I20260927 18:44:58.028153 2520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:58.029266 2520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:58.030498 2520 hybrid_clock.cc:648] HybridClock initialized: now 1790534698030436 us; error 60 us; skew 500 ppm
I20260927 18:44:58.030839 2520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:58 dist-test-slave-rfd5 krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534698, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:58.034219 2520 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:44:58.035807 2520 webserver.cc:460] Webserver started at http://127.31.26.193:39561/ using document root <none> and password file <none>
I20260927 18:44:58.036561 2520 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:58.036711 2520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:58.037030 2520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:58.040047 2520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "84c074d15d094e42baccf200c5232cd6"
format_stamp: "Formatted at 2026-09-27 18:44:58 on dist-test-slave-rfd5"
server_key: "e617ddd37d2aad1cc3519698c9b88279"
server_key_iv: "3d680312018b9e367e77700fa8078629"
server_key_version: "encryptionkey@0"
I20260927 18:44:58.040831 2520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "84c074d15d094e42baccf200c5232cd6"
format_stamp: "Formatted at 2026-09-27 18:44:58 on dist-test-slave-rfd5"
server_key: "e617ddd37d2aad1cc3519698c9b88279"
server_key_iv: "3d680312018b9e367e77700fa8078629"
server_key_version: "encryptionkey@0"
I20260927 18:44:58.046159 2520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 18:44:58.049892 2535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:58.051326 2520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:58.051517 2520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "84c074d15d094e42baccf200c5232cd6"
format_stamp: "Formatted at 2026-09-27 18:44:58 on dist-test-slave-rfd5"
server_key: "e617ddd37d2aad1cc3519698c9b88279"
server_key_iv: "3d680312018b9e367e77700fa8078629"
server_key_version: "encryptionkey@0"
I20260927 18:44:58.051730 2520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:58.076536 2520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:58.080718 2520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:58.081025 2520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:58.081996 2520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:58.083328 2520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:58.083431 2520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:58.083580 2520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:58.083653 2520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:58.118422 2520 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:36983
I20260927 18:44:58.118465 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:36983 every 8 connection(s)
I20260927 18:44:58.119809 2520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:44:58.129401 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2520
I20260927 18:44:58.129626 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:44:58.130448 31851 external_mini_cluster.cc:1428] Setting key cc3df7f957008736e97bbcb2e392a853
Sep 27 18:44:58 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534698, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:58.142527 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.122492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:53033 (local address 127.31.26.254:41605)
0927 18:44:58.122971 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:58.122978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:58.124110 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 18:44:58.126072 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:58.126077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:58.126147 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:58.126252 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:58.127985 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.128786 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.131346 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.131916 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.132053 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:58.136572 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:58.136577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:58.136581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:58.136617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:58.139339 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:58.140141 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:58.140146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:58.140148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:58.140269 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:58.140727 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:58.140739 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:58.140741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:58.141003 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:58.141181 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:58.141931 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 18:44:58.142162 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260927 18:44:58.143594 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.123164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41605 (local address 127.31.26.193:53033)
0927 18:44:58.123915 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:58.123953 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:58.125773 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:58.126475 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:58.126485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:58.127025 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:58.127737 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:58.127757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.128968 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.128984 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:58.131173 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:58.131186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.132142 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.132147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:58.132460 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:58.133280 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:58.133316 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:58.136294 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:58.139507 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:58.139518 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:58.139539 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:58.139996 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:58.140391 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:58.140396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:58.140399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:58.140603 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:58.141148 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:58.141156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:58.141536 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 18:44:58.141816 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 18:44:58.142407 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
I20260927 18:44:58.145354 2648 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41605
I20260927 18:44:58.145759 2648 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:58.146780 2648 heartbeater.cc:511] Master 127.31.26.254:41605 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:44:58.149469 2459 ts_manager.cc:194] Registered new tserver with Master: 84c074d15d094e42baccf200c5232cd6 (127.31.26.193:36983)
I20260927 18:44:58.151067 2459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:53033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:44:58.175837 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41605
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:58.503849 2654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:58.504202 2654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:58.504367 2654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:58.511714 2654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:58.511934 2654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:58.512118 2654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:44:58.522131 2654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:58.523577 2654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:58.525032 2654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:58.538621 2659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:58.538740 2660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:58.538694 2662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:58.539666 2654 server_base.cc:1034] running on GCE node
I20260927 18:44:58.540335 2654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:58.541450 2654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:58.542696 2654 hybrid_clock.cc:648] HybridClock initialized: now 1790534698542645 us; error 45 us; skew 500 ppm
I20260927 18:44:58.543061 2654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:58 dist-test-slave-rfd5 krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534698, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:58.546444 2654 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:44:58.548022 2654 webserver.cc:460] Webserver started at http://127.31.26.194:38147/ using document root <none> and password file <none>
I20260927 18:44:58.548831 2654 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:58.549014 2654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:58.549371 2654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:58.552371 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "39ac5812a23745d381d7be3f5d81e47f"
format_stamp: "Formatted at 2026-09-27 18:44:58 on dist-test-slave-rfd5"
server_key: "4268e7f0304cba95e7f4251b7dc72f1f"
server_key_iv: "f47c43ed0d4abc4a07a72c9489b17bca"
server_key_version: "encryptionkey@0"
I20260927 18:44:58.553267 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "39ac5812a23745d381d7be3f5d81e47f"
format_stamp: "Formatted at 2026-09-27 18:44:58 on dist-test-slave-rfd5"
server_key: "4268e7f0304cba95e7f4251b7dc72f1f"
server_key_iv: "f47c43ed0d4abc4a07a72c9489b17bca"
server_key_version: "encryptionkey@0"
I20260927 18:44:58.558898 2654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:44:58.562453 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:58.563892 2654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:58.564086 2654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "39ac5812a23745d381d7be3f5d81e47f"
format_stamp: "Formatted at 2026-09-27 18:44:58 on dist-test-slave-rfd5"
server_key: "4268e7f0304cba95e7f4251b7dc72f1f"
server_key_iv: "f47c43ed0d4abc4a07a72c9489b17bca"
server_key_version: "encryptionkey@0"
I20260927 18:44:58.564301 2654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:58.581758 2654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:58.585870 2654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:58.586185 2654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:58.587182 2654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:58.588562 2654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:58.588667 2654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:58.588816 2654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:58.588889 2654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:58.621838 2654 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:34957
I20260927 18:44:58.621881 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:34957 every 8 connection(s)
I20260927 18:44:58.623147 2654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:44:58.624727 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2654
I20260927 18:44:58.624905 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:44:58.625509 31851 external_mini_cluster.cc:1428] Setting key 6842cdda1a6690bfcdde0f3157ed0535
Sep 27 18:44:58 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534698, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:44:58.646152 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.626113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:43057 (local address 127.31.26.254:41605)
0927 18:44:58.626308 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:58.626314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:58.627681 (+ 1367us) server_negotiation.cc:407] Connection header received
0927 18:44:58.629619 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:58.629627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:58.629718 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:58.629831 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:58.631311 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.632515 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.634774 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.635181 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.635280 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:58.639871 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:58.639876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:58.639880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:58.639912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:58.642544 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:58.643537 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:58.643541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:58.643543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:58.643663 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:58.644195 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:58.644198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:58.644200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:58.644502 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:58.644637 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:58.645680 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0927 18:44:58.645880 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:44:58.647347 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:58.626701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41605 (local address 127.31.26.194:43057)
0927 18:44:58.627512 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:58.627564 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:58.629350 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:58.630041 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:58.630049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:58.630561 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:58.631134 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:58.631153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.632689 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.632701 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:58.634608 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:58.634626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:58.635423 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:58.635432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:58.635774 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:58.636541 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:58.636576 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:58.639625 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:58.642702 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:58.642714 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:58.642736 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:58.643405 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:58.643778 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:58.643785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:58.643788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:58.644082 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:58.644711 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:58.644720 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:58.645258 (+ 538us) client_negotiation.cc:769] Sending connection context
0927 18:44:58.645593 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 18:44:58.646119 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":342,"threads_started":1}
I20260927 18:44:58.649017 2782 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41605
I20260927 18:44:58.649362 2782 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:58.650139 2782 heartbeater.cc:511] Master 127.31.26.254:41605 requested a full tablet report, sending...
I20260927 18:44:58.651760 2459 ts_manager.cc:194] Registered new tserver with Master: 39ac5812a23745d381d7be3f5d81e47f (127.31.26.194:34957)
I20260927 18:44:58.652819 2459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:43057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:44:58.671545 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41605
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:59.005861 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:59.006237 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:59.006417 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:59.013896 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:59.014070 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:59.014253 2787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:44:59.024514 2787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:59.025920 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:59.027217 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:59.040344 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:59.040518 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:59.040529 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:59.041603 2787 server_base.cc:1034] running on GCE node
I20260927 18:44:59.042183 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:59.043177 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:59.044411 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790534699044378 us; error 62 us; skew 500 ppm
I20260927 18:44:59.044768 2787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:59.048122 2787 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:44:59.050019 2787 webserver.cc:460] Webserver started at http://127.31.26.195:33921/ using document root <none> and password file <none>
I20260927 18:44:59.050773 2787 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:59.050931 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:59.051254 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:59.054428 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "8138ab25e6a24560afd3b6af24610942"
format_stamp: "Formatted at 2026-09-27 18:44:59 on dist-test-slave-rfd5"
server_key: "341384eef8b367d4ffb0832e6cdeecf1"
server_key_iv: "d9ef5feb2f0e4989376536be0af9e226"
server_key_version: "encryptionkey@0"
I20260927 18:44:59.055338 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "8138ab25e6a24560afd3b6af24610942"
format_stamp: "Formatted at 2026-09-27 18:44:59 on dist-test-slave-rfd5"
server_key: "341384eef8b367d4ffb0832e6cdeecf1"
server_key_iv: "d9ef5feb2f0e4989376536be0af9e226"
server_key_version: "encryptionkey@0"
I20260927 18:44:59.061151 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:44:59.064738 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:59.066254 2787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:44:59.066452 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "8138ab25e6a24560afd3b6af24610942"
format_stamp: "Formatted at 2026-09-27 18:44:59 on dist-test-slave-rfd5"
server_key: "341384eef8b367d4ffb0832e6cdeecf1"
server_key_iv: "d9ef5feb2f0e4989376536be0af9e226"
server_key_version: "encryptionkey@0"
I20260927 18:44:59.066686 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:59.091377 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:59.095621 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:59.095940 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:59.096933 2787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:44:59.098398 2787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:44:59.098513 2787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:59.098661 2787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:44:59.098733 2787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:59.136801 2787 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35991
I20260927 18:44:59.136891 2914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35991 every 8 connection(s)
I20260927 18:44:59.138181 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:44:59.141425 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2787
I20260927 18:44:59.141607 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:44:59.142294 31851 external_mini_cluster.cc:1428] Setting key 1e39aec4d2994dfed59aa90446f4c6db
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:59.154273 2648 heartbeater.cc:503] Master 127.31.26.254:41605 was elected leader, sending a full tablet report...
I20260927 18:44:59.159765 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.140834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:55617 (local address 127.31.26.254:41605)
0927 18:44:59.141037 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:59.141043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:59.142599 (+ 1556us) server_negotiation.cc:407] Connection header received
0927 18:44:59.145032 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:59.145037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:59.145182 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:59.145303 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:59.146941 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.147705 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.149412 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.149823 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.149959 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:59.153542 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:59.153548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:59.153559 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:59.153594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:59.156634 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.157434 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.157440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.157445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.157578 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.158020 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.158023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.158025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.158279 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:59.158432 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:59.159143 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 18:44:59.159420 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:44:59.160748 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.141460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41605 (local address 127.31.26.195:55617)
0927 18:44:59.142383 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:59.142440 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:59.144716 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:59.145608 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:59.145617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:59.146147 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:59.146766 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:59.146785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.147858 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.147863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:59.149270 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:59.149284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.150057 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.150065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:59.150311 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:59.150940 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:59.150962 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:59.153299 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:59.156771 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:59.156781 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:59.156798 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:59.157309 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:59.157690 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:59.157696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:59.157700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:59.157885 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:59.158407 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:59.158415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:59.158766 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 18:44:59.159043 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 18:44:59.159607 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":343,"threads_started":1}
I20260927 18:44:59.162170 2915 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41605
I20260927 18:44:59.162494 2915 heartbeater.cc:464] Registering TS with master...
I20260927 18:44:59.163213 2915 heartbeater.cc:511] Master 127.31.26.254:41605 requested a full tablet report, sending...
I20260927 18:44:59.164603 2459 ts_manager.cc:194] Registered new tserver with Master: 8138ab25e6a24560afd3b6af24610942 (127.31.26.195:35991)
I20260927 18:44:59.165508 2459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:55617
I20260927 18:44:59.170258 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:44:59.217183 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.203260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.31.26.254:41605)
0927 18:44:59.203445 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:59.203450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:59.203962 (+ 512us) server_negotiation.cc:407] Connection header received
0927 18:44:59.204113 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:59.204117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:59.204178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:59.204272 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:59.205670 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.206375 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.207567 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.207953 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.208053 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:59.212067 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:59.212072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:59.212076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:59.212112 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:59.214831 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.215454 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.215458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.215460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.215602 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.215953 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.215956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.215958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.216273 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:59.216452 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:59.216731 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 18:44:59.216888 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 18:44:59.217713 2459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40730
I20260927 18:44:59.219650 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2520
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:44:59.235090 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.220575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.31.26.193:36983)
0927 18:44:59.221226 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:59.221234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:59.221258 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:44:59.221310 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:59.221316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:59.221533 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:59.221662 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:59.222776 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.223940 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.225067 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.225652 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.225748 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:59.229814 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:59.229826 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:59.229841 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:59.229873 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:59.232437 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.233035 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.233042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.233044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.233229 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.233554 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.233557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.233559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.233879 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:59.234112 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:59.234429 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 18:44:59.234692 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
W20260927 18:44:59.236313 2643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52622
W20260927 18:44:59.237507 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:59.237694 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2520
I20260927 18:44:59.253532 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2654
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:44:59.269997 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.254410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.31.26.194:34957)
0927 18:44:59.255012 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:59.255019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:59.255046 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:44:59.255116 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:59.255122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:59.255331 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:59.255476 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:59.256781 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.257879 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.259098 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.259737 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.259836 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:59.264140 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:59.264153 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:59.264170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:59.264207 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:59.267055 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.267722 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.267729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.267731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.267872 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.268322 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.268325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.268327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.268706 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:59.268950 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:59.269488 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 18:44:59.269690 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":327,"threads_started":1}
W20260927 18:44:59.271375 2777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46892
W20260927 18:44:59.272637 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:59.272789 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2654
I20260927 18:44:59.288059 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2787
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534699, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:44:59.305541 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:59.289200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.31.26.195:35991)
0927 18:44:59.289900 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:59.289909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:59.289948 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:44:59.290008 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:59.290016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:59.290235 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:59.290363 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:59.291700 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.292740 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.294083 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:59.294660 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:59.294763 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:59.299520 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:59.299534 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:59.299551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:59.299597 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:59.302682 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.303415 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.303423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.303426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.303546 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:59.303919 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:59.303928 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:59.303930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:59.304275 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:59.304583 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:59.304896 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 18:44:59.305198 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":365,"threads_started":1}
W20260927 18:44:59.306964 2910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55628
W20260927 18:44:59.308158 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:59.308315 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2787
I20260927 18:44:59.323477 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2427
I20260927 18:44:59.324973 2489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:59.429224 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2427
2026-09-27T18:44:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3657 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2932](info): set up 2 sockets
Sep 27 18:44:59 dist-test-slave-rfd5 krb5kdc[2932](info): commencing operation
krb5kdc: starting...
W20260927 18:45:00.517539 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:00 dist-test-slave-rfd5 krb5kdc[2932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:45:00.716897 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:39687
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:01.037451 2945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:01.037750 2945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:01.037837 2945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:01.044891 2945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:01.044998 2945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:01.045044 2945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:01.045156 2945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:01.045217 2945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:01.055063 2945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:01.056375 2945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:01.057757 2945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:01.070343 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.070231 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.070501 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.071480 2945 server_base.cc:1034] running on GCE node
I20260927 18:45:01.072330 2945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:01.073979 2945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:01.075234 2945 hybrid_clock.cc:648] HybridClock initialized: now 1790534701075169 us; error 62 us; skew 500 ppm
I20260927 18:45:01.075613 2945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:01 dist-test-slave-rfd5 krb5kdc[2932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:01.086409 2945 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:45:01.088081 2945 webserver.cc:460] Webserver started at http://127.31.26.254:46847/ using document root <none> and password file <none>
I20260927 18:45:01.088794 2945 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:01.088896 2945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:01.089246 2945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:01.092221 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "c20ee06612804d1b803ce2588f08a2ae"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-rfd5"
server_key: "18686f330f831041ab49d62b76ad0e30"
server_key_iv: "8b802cc88e05aeeb739cefd09a4a0e69"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.092967 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "c20ee06612804d1b803ce2588f08a2ae"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-rfd5"
server_key: "18686f330f831041ab49d62b76ad0e30"
server_key_iv: "8b802cc88e05aeeb739cefd09a4a0e69"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.098234 2945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:45:01.101878 2960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.103297 2945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:01.103525 2945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "c20ee06612804d1b803ce2588f08a2ae"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-rfd5"
server_key: "18686f330f831041ab49d62b76ad0e30"
server_key_iv: "8b802cc88e05aeeb739cefd09a4a0e69"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.103730 2945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:01.117684 2945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:01.122341 2945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:01.122649 2945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:01.141431 2945 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:39687
I20260927 18:45:01.141435 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:39687 every 8 connection(s)
I20260927 18:45:01.142886 2945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:01.145754 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 2945
I20260927 18:45:01.146016 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:01.146827 31851 external_mini_cluster.cc:1428] Setting key 3242451925a93a6b8163fc015c87241a
I20260927 18:45:01.147255 3012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:01.161964 3012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae: Bootstrap starting.
Sep 27 18:45:01 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:01.166090 3012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:01.167476 3012 log.cc:824] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:01.170951 3012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae: No bootstrap required, opened a new log
I20260927 18:45:01.172096 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.148923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.31.26.254:39687)
0927 18:45:01.149785 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.149797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.149837 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:45:01.151935 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.151957 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.152370 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.152751 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.154583 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.155872 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.157696 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.158425 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.158574 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.163211 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.163222 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.163241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.163295 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.167306 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.168110 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.168122 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.168131 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.168305 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.168731 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.168734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.168736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.169487 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.169781 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.170172 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.170596 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":449,"threads_started":1}
I20260927 18:45:01.179396 3012 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20ee06612804d1b803ce2588f08a2ae" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39687 } }
I20260927 18:45:01.179764 3012 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:01.179836 3012 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20ee06612804d1b803ce2588f08a2ae, State: Initialized, Role: FOLLOWER
I20260927 18:45:01.180320 3012 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20ee06612804d1b803ce2588f08a2ae" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39687 } }
I20260927 18:45:01.180506 3012 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:01.180598 3012 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:01.180716 3012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:01.182900 3012 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20ee06612804d1b803ce2588f08a2ae" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39687 } }
I20260927 18:45:01.183548 3012 leader_election.cc:302] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20ee06612804d1b803ce2588f08a2ae; no voters:
I20260927 18:45:01.184120 3012 leader_election.cc:288] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:01.184266 3017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:01.184978 3017 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [term 1 LEADER]: Becoming Leader. State: Replica: c20ee06612804d1b803ce2588f08a2ae, State: Running, Role: LEADER
I20260927 18:45:01.185566 3017 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20ee06612804d1b803ce2588f08a2ae" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39687 } }
I20260927 18:45:01.186208 3012 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:01.188264 3019 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20ee06612804d1b803ce2588f08a2ae. Latest consensus state: current_term: 1 leader_uuid: "c20ee06612804d1b803ce2588f08a2ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20ee06612804d1b803ce2588f08a2ae" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39687 } } }
I20260927 18:45:01.188500 3018 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20ee06612804d1b803ce2588f08a2ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20ee06612804d1b803ce2588f08a2ae" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39687 } } }
I20260927 18:45:01.188709 3019 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:01.188764 3018 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:01.189563 3022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:01.195264 3022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:01.204619 3022 catalog_manager.cc:1347] Generated new cluster ID: ab6b767b2f4d4c3eae6b9e60c932cfe4
I20260927 18:45:01.204749 3022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:01.225915 3022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:01.227840 3022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:01.234967 3022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c20ee06612804d1b803ce2588f08a2ae: Generated new TSK 0
I20260927 18:45:01.235845 3022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:45:01.282748 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39687
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:01.615044 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:01.615324 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:01.615418 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:01.622780 3038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:01.622920 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:01.623113 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:01.633152 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:01.634512 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:01.635845 3038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:01.649294 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.649343 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:01.649341 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:01.650173 3038 server_base.cc:1034] running on GCE node
I20260927 18:45:01.650885 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:01.651880 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:01.653183 3038 hybrid_clock.cc:648] HybridClock initialized: now 1790534701653118 us; error 52 us; skew 500 ppm
I20260927 18:45:01.653522 3038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:01 dist-test-slave-rfd5 krb5kdc[2932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:01.656690 3038 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:45:01.658206 3038 webserver.cc:460] Webserver started at http://127.31.26.193:41085/ using document root <none> and password file <none>
I20260927 18:45:01.658876 3038 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:01.658983 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:01.659238 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:01.662177 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "f73fa9211eff403cbaad7c3f2bccd7f4"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-rfd5"
server_key: "8f50e4b11f32fa7ef2748825f116f377"
server_key_iv: "46f1da64cedb71d969ffa406d3c8c150"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.662883 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "f73fa9211eff403cbaad7c3f2bccd7f4"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-rfd5"
server_key: "8f50e4b11f32fa7ef2748825f116f377"
server_key_iv: "46f1da64cedb71d969ffa406d3c8c150"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.668200 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:45:01.671904 3053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.673517 3038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:01.673686 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "f73fa9211eff403cbaad7c3f2bccd7f4"
format_stamp: "Formatted at 2026-09-27 18:45:01 on dist-test-slave-rfd5"
server_key: "8f50e4b11f32fa7ef2748825f116f377"
server_key_iv: "46f1da64cedb71d969ffa406d3c8c150"
server_key_version: "encryptionkey@0"
I20260927 18:45:01.673898 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:01.684599 3038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:01.689448 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:01.689752 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:01.690781 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:01.692312 3038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:01.692384 3038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.692574 3038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:01.692625 3038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:01.727025 3038 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:44417
I20260927 18:45:01.727072 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:44417 every 8 connection(s)
I20260927 18:45:01.728427 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:01.732731 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3038
I20260927 18:45:01.732976 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:01.733659 31851 external_mini_cluster.cc:1428] Setting key a57ace9b3518d054d85ea20fdb3cd95d
Sep 27 18:45:01 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534701, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:01.751673 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.731179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:44033 (local address 127.31.26.254:39687)
0927 18:45:01.731648 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:01.731653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:01.732677 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 18:45:01.735127 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:01.735135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:01.735247 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:01.735363 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:01.737180 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.737906 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.739530 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.739926 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.740063 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.744444 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:01.744450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:01.744453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:01.744489 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:01.748165 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.749062 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.749071 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.749078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.749264 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:01.749702 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:01.749710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:01.749717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:01.750152 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:01.750318 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:01.751002 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 18:45:01.751248 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
I20260927 18:45:01.752381 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:01.731648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:39687 (local address 127.31.26.193:44033)
0927 18:45:01.732496 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:01.732542 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:01.734770 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:01.735709 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:01.735723 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:01.736346 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:01.736953 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.736970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.738063 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.738068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.739373 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:01.739389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:01.740161 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:01.740169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:01.740426 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:01.741301 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:01.741359 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:01.744177 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:01.748338 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.748345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.748360 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.748910 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.749385 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:01.749388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:01.749391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:01.749576 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:01.750277 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:01.750283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:01.750607 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 18:45:01.750834 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 18:45:01.751247 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:45:01.754004 3166 heartbeater.cc:347] Connected to a master server at 127.31.26.254:39687
I20260927 18:45:01.754350 3166 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:45:01.755232 3166 heartbeater.cc:511] Master 127.31.26.254:39687 requested a full tablet report, sending...
I20260927 18:45:01.757691 2977 ts_manager.cc:194] Registered new tserver with Master: f73fa9211eff403cbaad7c3f2bccd7f4 (127.31.26.193:44417)
I20260927 18:45:01.759840 2977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:44033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:45:01.780795 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39687
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:02.112035 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:02.112550 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:02.112725 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:02.122062 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:02.122181 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:02.122356 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:02.132452 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:02.133863 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:02.135216 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:02.148765 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:02.148864 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:02.149405 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:02.149991 3172 server_base.cc:1034] running on GCE node
I20260927 18:45:02.150699 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:02.151814 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:02.153050 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790534702153011 us; error 41 us; skew 500 ppm
I20260927 18:45:02.153435 3172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:02.157076 3172 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:45:02.158676 3172 webserver.cc:460] Webserver started at http://127.31.26.194:45277/ using document root <none> and password file <none>
I20260927 18:45:02.159394 3172 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:02.159555 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:02.159881 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:02.162827 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "672e2c7162f94a1ab95c2b3040a894a2"
format_stamp: "Formatted at 2026-09-27 18:45:02 on dist-test-slave-rfd5"
server_key: "7f2d005b8057796e2a03a96e4e7d3532"
server_key_iv: "035c85ab592e6b13a572e08b2d443990"
server_key_version: "encryptionkey@0"
I20260927 18:45:02.163625 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "672e2c7162f94a1ab95c2b3040a894a2"
format_stamp: "Formatted at 2026-09-27 18:45:02 on dist-test-slave-rfd5"
server_key: "7f2d005b8057796e2a03a96e4e7d3532"
server_key_iv: "035c85ab592e6b13a572e08b2d443990"
server_key_version: "encryptionkey@0"
I20260927 18:45:02.168923 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:45:02.172643 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:02.174214 3172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:02.174414 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "672e2c7162f94a1ab95c2b3040a894a2"
format_stamp: "Formatted at 2026-09-27 18:45:02 on dist-test-slave-rfd5"
server_key: "7f2d005b8057796e2a03a96e4e7d3532"
server_key_iv: "035c85ab592e6b13a572e08b2d443990"
server_key_version: "encryptionkey@0"
I20260927 18:45:02.174614 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:02.198360 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:02.202899 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:02.203213 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:02.204244 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:02.205916 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:02.206069 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:45:02.206210 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:02.206275 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:02.239673 3172 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:46181
I20260927 18:45:02.239765 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:46181 every 8 connection(s)
I20260927 18:45:02.241022 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:02.251418 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3172
I20260927 18:45:02.251624 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:02.252079 31851 external_mini_cluster.cc:1428] Setting key 55072a71aa7d53440029834464571f18
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:02.262773 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.243961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:48507 (local address 127.31.26.254:39687)
0927 18:45:02.244205 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.244211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.245404 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 18:45:02.247319 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.247325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.247423 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.247538 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:02.249110 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.249823 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.251408 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.251960 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.252095 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.256345 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:02.256351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:02.256354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:02.256388 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:02.259293 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.260185 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.260191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.260197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.260346 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.260824 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.260831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.260834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.261190 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:02.261350 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.262129 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.262387 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:45:02.263875 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.244457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:39687 (local address 127.31.26.194:48507)
0927 18:45:02.245241 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:02.245280 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:02.247014 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:02.247789 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:02.247798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:02.248319 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:02.248915 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.248932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.249988 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.249994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.251255 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.251267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.252202 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.252209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.252450 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.253264 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:02.253295 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:02.256087 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:02.259493 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:02.259504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:02.259525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:02.260037 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:02.260475 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:02.260480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:02.260484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:02.260703 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:02.261336 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:02.261345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:02.261738 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 18:45:02.262048 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 18:45:02.262653 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":280,"threads_started":1}
I20260927 18:45:02.265679 3300 heartbeater.cc:347] Connected to a master server at 127.31.26.254:39687
I20260927 18:45:02.266021 3300 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:02.266913 3300 heartbeater.cc:511] Master 127.31.26.254:39687 requested a full tablet report, sending...
I20260927 18:45:02.268560 2977 ts_manager.cc:194] Registered new tserver with Master: 672e2c7162f94a1ab95c2b3040a894a2 (127.31.26.194:46181)
I20260927 18:45:02.269479 2977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:48507
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:45:02.296455 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39687
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:02.625558 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:02.625856 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:02.626019 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:02.633327 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:02.633442 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:02.633616 3305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:02.643446 3305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:02.644785 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:02.646106 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:02.659423 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:02.659392 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:02.659446 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:02.660331 3305 server_base.cc:1034] running on GCE node
I20260927 18:45:02.661008 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:02.661996 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:02.663221 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790534702663170 us; error 45 us; skew 500 ppm
I20260927 18:45:02.663549 3305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:02.666695 3305 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:45:02.668357 3305 webserver.cc:460] Webserver started at http://127.31.26.195:33757/ using document root <none> and password file <none>
I20260927 18:45:02.669062 3305 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:02.669255 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:02.669564 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:02.672350 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "1cf6d7843a694cba92a3c00cc04ee8ec"
format_stamp: "Formatted at 2026-09-27 18:45:02 on dist-test-slave-rfd5"
server_key: "ff74fe289b2eccec1478d1300723578a"
server_key_iv: "7a0d9dd1f1f36ab91d609bcf5903e4e2"
server_key_version: "encryptionkey@0"
I20260927 18:45:02.673197 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "1cf6d7843a694cba92a3c00cc04ee8ec"
format_stamp: "Formatted at 2026-09-27 18:45:02 on dist-test-slave-rfd5"
server_key: "ff74fe289b2eccec1478d1300723578a"
server_key_iv: "7a0d9dd1f1f36ab91d609bcf5903e4e2"
server_key_version: "encryptionkey@0"
I20260927 18:45:02.678350 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:45:02.681954 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:02.683218 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:02.683411 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "1cf6d7843a694cba92a3c00cc04ee8ec"
format_stamp: "Formatted at 2026-09-27 18:45:02 on dist-test-slave-rfd5"
server_key: "ff74fe289b2eccec1478d1300723578a"
server_key_iv: "7a0d9dd1f1f36ab91d609bcf5903e4e2"
server_key_version: "encryptionkey@0"
I20260927 18:45:02.683612 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:02.713385 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:02.717506 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:02.717813 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:02.718808 3305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:02.720152 3305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:02.720260 3305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:02.720418 3305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:02.720507 3305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:02.753742 3305 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:33151
I20260927 18:45:02.753791 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:33151 every 8 connection(s)
I20260927 18:45:02.755048 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:02.756460 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3305
I20260927 18:45:02.756665 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:02.757135 31851 external_mini_cluster.cc:1428] Setting key d55ed402b104e6c63e52fb1a2d097da0
I20260927 18:45:02.763012 3166 heartbeater.cc:503] Master 127.31.26.254:39687 was elected leader, sending a full tablet report...
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:02.776558 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.757768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:37303 (local address 127.31.26.254:39687)
0927 18:45:02.757961 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.757967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.759465 (+ 1498us) server_negotiation.cc:407] Connection header received
0927 18:45:02.761560 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.761564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.761630 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.761713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:02.763414 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.764252 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.766043 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.766465 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.766591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.770169 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:02.770176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:02.770180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:02.770216 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:02.773651 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.774407 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.774412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.774414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.774544 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.775005 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.775008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.775010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.775305 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:02.775417 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.776079 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.776244 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:45:02.777621 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.758244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:39687 (local address 127.31.26.195:37303)
0927 18:45:02.759290 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:02.759343 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:02.761291 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:02.761894 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:02.761903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:02.762416 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:02.763174 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.763200 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.764439 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.764444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.765893 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:02.765905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.766701 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.766708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:02.766949 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.767624 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:02.767647 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:02.769912 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:02.773793 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:02.773800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:02.773814 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:02.774263 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:02.774638 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:02.774641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:02.774644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:02.774897 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:02.775412 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:02.775417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:02.775742 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 18:45:02.775974 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 18:45:02.776370 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":585,"threads_started":1}
I20260927 18:45:02.779253 3433 heartbeater.cc:347] Connected to a master server at 127.31.26.254:39687
I20260927 18:45:02.779641 3433 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:02.780433 3433 heartbeater.cc:511] Master 127.31.26.254:39687 requested a full tablet report, sending...
I20260927 18:45:02.782001 2977 ts_manager.cc:194] Registered new tserver with Master: 1cf6d7843a694cba92a3c00cc04ee8ec (127.31.26.195:33151)
I20260927 18:45:02.782850 2977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:37303
I20260927 18:45:02.785270 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:45:02.833374 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.817696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.31.26.193:44417)
0927 18:45:02.818409 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.818417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.818446 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:45:02.818610 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.818617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.818841 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.818973 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:02.820344 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.821422 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.822627 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.823205 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.823311 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.827399 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:02.827413 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:02.827430 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:02.827473 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:02.830481 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.831194 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.831202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.831204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.831370 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.831823 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.831827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.831829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.832180 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:02.832440 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.832717 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.832919 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":285,"threads_started":1}
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:02.852231 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.837898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.31.26.254:39687)
0927 18:45:02.838145 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.838152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.838386 (+ 234us) server_negotiation.cc:407] Connection header received
0927 18:45:02.838614 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.838622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.838688 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.838793 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:02.840070 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.840800 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.842176 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.842612 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.842760 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.847063 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:02.847069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:02.847073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:02.847114 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:02.849835 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.850579 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.850583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.850586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.850716 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.851051 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.851055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.851056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.851391 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:02.851565 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.851835 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.852003 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:45:02.852859 2977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55250
I20260927 18:45:02.853976 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3038
I20260927 18:45:02.866014 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.854949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.31.26.193:44417)
0927 18:45:02.855120 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.855125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.855364 (+ 239us) server_negotiation.cc:407] Connection header received
0927 18:45:02.855490 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.855493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.855555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.855632 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:02.856710 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.857450 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.858591 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.859009 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.859097 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.861432 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:02.861437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:02.861440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:02.861469 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:02.863871 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.864452 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.864456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.864458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.864575 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.864880 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.864883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.864885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.865195 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:02.865364 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.865612 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.865773 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:45:02.866663 3161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:02.949059 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3038
I20260927 18:45:02.965710 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3172
Sep 27 18:45:02 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:45:02.982525 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:02.966742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.31.26.194:46181)
0927 18:45:02.967282 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:02.967289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:02.967314 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:45:02.967380 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:02.967386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:02.967597 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:02.967769 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:02.968994 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.970082 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.971454 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:02.972028 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:02.972177 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:02.976605 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:02.976619 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:02.976634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:02.976670 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:02.979578 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.980311 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.980318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.980320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.980443 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:02.980835 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:02.980846 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:02.980849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:02.981253 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:02.981576 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:02.981894 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 18:45:02.982169 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260927 18:45:02.983774 3295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:03.067276 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3172
I20260927 18:45:03.083792 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3305
Sep 27 18:45:03 dist-test-slave-rfd5 krb5kdc[2932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:45:03.101567 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:03.084841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.31.26.195:33151)
0927 18:45:03.085494 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:03.085501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:03.085529 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:45:03.085587 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:03.085593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:03.085801 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:03.085981 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:03.087263 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:03.088456 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:03.089860 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:03.090718 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:03.090854 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:03.096037 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:03.096051 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:03.096067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:03.096107 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:03.098891 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:03.099568 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:03.099575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:03.099577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:03.099701 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:03.100024 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:03.100027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:03.100029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:03.100376 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:03.100612 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:03.100938 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:45:03.101257 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":374,"threads_started":1}
I20260927 18:45:03.102870 3428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:03.186383 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3305
I20260927 18:45:03.202445 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 2945
I20260927 18:45:03.203714 3007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:03.305274 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 2945
2026-09-27T18:45:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3875 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:03 dist-test-slave-rfd5 krb5kdc[3452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:03 dist-test-slave-rfd5 krb5kdc[3452](info): set up 2 sockets
Sep 27 18:45:03 dist-test-slave-rfd5 krb5kdc[3452](info): commencing operation
krb5kdc: starting...
W20260927 18:45:04.457538 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:04 dist-test-slave-rfd5 krb5kdc[3452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:45:04.652459 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35753
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:04.978117 3465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:04.978425 3465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:04.978592 3465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:04.985803 3465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:04.985910 3465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:04.986007 3465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:04.986121 3465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:04.986187 3465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:04.995975 3465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:04.997332 3465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:04.998658 3465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:05.010097 3470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:05.010409 3473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:05.010067 3471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:05.011062 3465 server_base.cc:1034] running on GCE node
I20260927 18:45:05.011960 3465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:05.013476 3465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:05.014715 3465 hybrid_clock.cc:648] HybridClock initialized: now 1790534705014674 us; error 39 us; skew 500 ppm
I20260927 18:45:05.015105 3465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:05 dist-test-slave-rfd5 krb5kdc[3452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534705, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:05.018249 3465 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:45:05.019416 3465 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:05.019548 3465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:05.019868 3465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:05.022879 3465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "d552ec3dce3743a59de23a53f02b89e9"
format_stamp: "Formatted at 2026-09-27 18:45:05 on dist-test-slave-rfd5"
server_key: "a8b7466c8b9c20a9a4c927b8bfa4423e"
server_key_iv: "2925f3e80be4ab47d35fb153960e24f8"
server_key_version: "encryptionkey@0"
I20260927 18:45:05.023672 3465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "d552ec3dce3743a59de23a53f02b89e9"
format_stamp: "Formatted at 2026-09-27 18:45:05 on dist-test-slave-rfd5"
server_key: "a8b7466c8b9c20a9a4c927b8bfa4423e"
server_key_iv: "2925f3e80be4ab47d35fb153960e24f8"
server_key_version: "encryptionkey@0"
I20260927 18:45:05.028595 3465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:05.032110 3479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:05.033545 3465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:05.033743 3465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "d552ec3dce3743a59de23a53f02b89e9"
format_stamp: "Formatted at 2026-09-27 18:45:05 on dist-test-slave-rfd5"
server_key: "a8b7466c8b9c20a9a4c927b8bfa4423e"
server_key_iv: "2925f3e80be4ab47d35fb153960e24f8"
server_key_version: "encryptionkey@0"
I20260927 18:45:05.033993 3465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:05.048722 3465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:05.052736 3465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:05.053014 3465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:05.071094 3465 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:35753
I20260927 18:45:05.071055 3530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:35753 every 8 connection(s)
I20260927 18:45:05.071969 3465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:05.075599 3531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:05.081453 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3465
I20260927 18:45:05.081693 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:05.082333 31851 external_mini_cluster.cc:1428] Setting key 829d6c46a1b60a838ee30d92958e6814
I20260927 18:45:05.086580 3531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9: Bootstrap starting.
I20260927 18:45:05.090377 3531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:05.091552 3531 log.cc:824] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:05.094770 3531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9: No bootstrap required, opened a new log
Sep 27 18:45:05 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:05.104852 3531 raft_consensus.cc:348] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d552ec3dce3743a59de23a53f02b89e9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35753 } }
I20260927 18:45:05.105374 3531 raft_consensus.cc:374] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:05.105525 3531 raft_consensus.cc:729] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d552ec3dce3743a59de23a53f02b89e9, State: Initialized, Role: FOLLOWER
I20260927 18:45:05.106284 3531 consensus_queue.cc:260] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d552ec3dce3743a59de23a53f02b89e9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35753 } }
I20260927 18:45:05.106600 3531 raft_consensus.cc:388] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:05.106813 3531 raft_consensus.cc:482] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:05.107036 3531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:05.107955 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:05.084426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.31.26.254:35753)
0927 18:45:05.085243 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:05.085254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:05.085293 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:45:05.087049 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:05.087071 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:05.087466 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:05.087786 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:05.089753 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:05.091142 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:05.092846 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:05.093587 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:05.093730 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:05.098710 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:05.098723 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:05.098747 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:05.098813 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:05.103178 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:05.104053 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:05.104063 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:05.104074 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:05.104280 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:05.104707 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:05.104713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:05.104716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:05.105450 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:05.105780 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:05.106241 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 18:45:05.106667 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":376,"threads_started":1}
I20260927 18:45:05.109757 3531 raft_consensus.cc:504] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d552ec3dce3743a59de23a53f02b89e9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35753 } }
I20260927 18:45:05.110380 3531 leader_election.cc:302] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d552ec3dce3743a59de23a53f02b89e9; no voters:
I20260927 18:45:05.111109 3531 leader_election.cc:288] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:05.111225 3536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:05.111979 3536 raft_consensus.cc:686] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [term 1 LEADER]: Becoming Leader. State: Replica: d552ec3dce3743a59de23a53f02b89e9, State: Running, Role: LEADER
I20260927 18:45:05.112649 3536 consensus_queue.cc:237] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d552ec3dce3743a59de23a53f02b89e9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35753 } }
I20260927 18:45:05.113739 3531 sys_catalog.cc:564] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:05.115846 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d552ec3dce3743a59de23a53f02b89e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d552ec3dce3743a59de23a53f02b89e9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35753 } } }
I20260927 18:45:05.115986 3537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d552ec3dce3743a59de23a53f02b89e9. Latest consensus state: current_term: 1 leader_uuid: "d552ec3dce3743a59de23a53f02b89e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d552ec3dce3743a59de23a53f02b89e9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35753 } } }
I20260927 18:45:05.116382 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:05.116456 3537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:05.122457 3541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:05.127759 3541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:05.137176 3541 catalog_manager.cc:1347] Generated new cluster ID: 41553a3a1c7743f0b27a6f4d785971df
I20260927 18:45:05.137351 3541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:05.164650 3541 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:05.166729 3541 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:05.176678 3541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9: Generated new TSK 0
I20260927 18:45:05.177793 3541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:45:05.240756 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35753
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:05.569067 3557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:05.569380 3557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:05.569479 3557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:05.576633 3557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:05.576757 3557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:05.576879 3557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:05.586737 3557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:05.588047 3557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:05.589466 3557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:05.602382 3563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:05.602448 3562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:05.602406 3565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:05.603328 3557 server_base.cc:1034] running on GCE node
I20260927 18:45:05.604038 3557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:05.605100 3557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:05.606343 3557 hybrid_clock.cc:648] HybridClock initialized: now 1790534705606262 us; error 72 us; skew 500 ppm
I20260927 18:45:05.606698 3557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:05 dist-test-slave-rfd5 krb5kdc[3452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534705, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:05.610280 3557 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:45:05.611480 3557 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:05.611593 3557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:05.611946 3557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:05.614976 3557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c"
format_stamp: "Formatted at 2026-09-27 18:45:05 on dist-test-slave-rfd5"
server_key: "a3440a41ccdfa996c3630da949ad0e9c"
server_key_iv: "b5bb4919715e0d8469986241a93782d0"
server_key_version: "encryptionkey@0"
I20260927 18:45:05.615788 3557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c"
format_stamp: "Formatted at 2026-09-27 18:45:05 on dist-test-slave-rfd5"
server_key: "a3440a41ccdfa996c3630da949ad0e9c"
server_key_iv: "b5bb4919715e0d8469986241a93782d0"
server_key_version: "encryptionkey@0"
I20260927 18:45:05.621348 3557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:45:05.625231 3571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:05.626917 3557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:05.627141 3557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c"
format_stamp: "Formatted at 2026-09-27 18:45:05 on dist-test-slave-rfd5"
server_key: "a3440a41ccdfa996c3630da949ad0e9c"
server_key_iv: "b5bb4919715e0d8469986241a93782d0"
server_key_version: "encryptionkey@0"
I20260927 18:45:05.627403 3557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:05.639585 3557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:05.644233 3557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:05.644534 3557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:05.645641 3557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:05.647074 3557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:05.647162 3557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:05.647315 3557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:05.647411 3557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:05.682484 3557 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:45421
I20260927 18:45:05.682542 3683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:45421 every 8 connection(s)
I20260927 18:45:05.683418 3557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:05.690368 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3557
I20260927 18:45:05.690623 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:05.691306 31851 external_mini_cluster.cc:1428] Setting key 896e206be6f583bce9492783638724b6
Sep 27 18:45:05 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534705, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:05.706971 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:05.686133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:34753 (local address 127.31.26.254:35753)
0927 18:45:05.686618 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:05.686623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:05.687697 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 18:45:05.689713 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:05.689719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:05.689842 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:05.689979 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:05.691899 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:05.692772 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:05.694543 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:05.694967 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:05.695120 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:05.699140 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:05.699146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:05.699149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:05.699185 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:05.703004 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:05.704033 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:05.704042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:05.704048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:05.704203 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:05.704738 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:05.704746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:05.704752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:05.705180 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:05.705378 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:05.706195 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 18:45:05.706488 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260927 18:45:05.707702 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:05.686739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35753 (local address 127.31.26.193:34753)
0927 18:45:05.687500 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:05.687550 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:05.689396 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:05.690256 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:05.690265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:05.690959 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:05.691706 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:05.691731 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:05.692966 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:05.692970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:05.694370 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:05.694382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:05.695237 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:05.695245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:05.695494 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:05.696302 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:05.696327 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:05.698845 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:05.703260 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:05.703269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:05.703286 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:05.703868 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:05.704370 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:05.704374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:05.704376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:05.704603 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:05.705364 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:05.705371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:05.705762 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 18:45:05.706025 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 18:45:05.706463 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":288,"threads_started":1}
I20260927 18:45:05.709486 3684 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35753
I20260927 18:45:05.709882 3684 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:05.710788 3684 heartbeater.cc:511] Master 127.31.26.254:35753 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:45:05.713477 3496 ts_manager.cc:194] Registered new tserver with Master: b9ce013ceebc40f1bcb5de95ec0f5c5c (127.31.26.193:45421)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:45:05.715232 3496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:34753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:45:05.740811 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35753
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:06.076850 3690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:06.077229 3690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:06.077414 3690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:06.085558 3690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:06.085675 3690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:06.085847 3690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:06.096156 3690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:06.097657 3690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:06.098997 3690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:06.113842 3696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:06.113793 3695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:06.114581 3698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:06.115033 3690 server_base.cc:1034] running on GCE node
I20260927 18:45:06.115769 3690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:06.117039 3690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:06.118314 3690 hybrid_clock.cc:648] HybridClock initialized: now 1790534706118234 us; error 64 us; skew 500 ppm
I20260927 18:45:06.118752 3690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:06 dist-test-slave-rfd5 krb5kdc[3452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:06.127619 3690 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:45:06.129006 3690 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:06.129199 3690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:06.129541 3690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:06.132761 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba"
format_stamp: "Formatted at 2026-09-27 18:45:06 on dist-test-slave-rfd5"
server_key: "5c4a6f5ad0becbc97ae8ace1b78f8bd0"
server_key_iv: "c4aa69b7510daf0f78c54aa6cfd94178"
server_key_version: "encryptionkey@0"
I20260927 18:45:06.133731 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba"
format_stamp: "Formatted at 2026-09-27 18:45:06 on dist-test-slave-rfd5"
server_key: "5c4a6f5ad0becbc97ae8ace1b78f8bd0"
server_key_iv: "c4aa69b7510daf0f78c54aa6cfd94178"
server_key_version: "encryptionkey@0"
I20260927 18:45:06.139328 3690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:45:06.143316 3704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:06.144799 3690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:06.144994 3690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba"
format_stamp: "Formatted at 2026-09-27 18:45:06 on dist-test-slave-rfd5"
server_key: "5c4a6f5ad0becbc97ae8ace1b78f8bd0"
server_key_iv: "c4aa69b7510daf0f78c54aa6cfd94178"
server_key_version: "encryptionkey@0"
I20260927 18:45:06.145255 3690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:06.170099 3690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:06.174497 3690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:06.174785 3690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:06.175913 3690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:06.177357 3690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:06.177481 3690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:06.177611 3690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:06.177700 3690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:06.211028 3690 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:45521
I20260927 18:45:06.211114 3816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:45521 every 8 connection(s)
I20260927 18:45:06.211913 3690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:06.221820 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3690
I20260927 18:45:06.222036 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:06.222462 31851 external_mini_cluster.cc:1428] Setting key 76604570fa94e1e350c286cb9da5a1fa
Sep 27 18:45:06 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:06.233230 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.214531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:46457 (local address 127.31.26.254:35753)
0927 18:45:06.214956 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:06.214960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:06.215913 (+ 953us) server_negotiation.cc:407] Connection header received
0927 18:45:06.217824 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:06.217830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:06.217928 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:06.218044 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:06.219566 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.220267 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.221926 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.222447 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.222582 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.226744 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:06.226749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:06.226753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:06.226784 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:06.229513 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:06.230349 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:06.230357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:06.230391 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:06.230552 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:06.230988 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:06.230996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:06.231002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:06.231389 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:06.231538 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:06.232464 (+ 926us) server_negotiation.cc:299] Negotiation successful
0927 18:45:06.232727 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260927 18:45:06.234093 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.215037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35753 (local address 127.31.26.194:46457)
0927 18:45:06.215731 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:06.215773 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:06.217552 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:06.218274 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:06.218282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:06.218821 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:06.219392 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.219410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.220419 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.220431 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.221775 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.221788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.222670 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.222677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.222900 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.223592 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:06.223614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:06.226493 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:06.229661 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:06.229674 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:06.229696 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:06.230209 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:06.230669 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:06.230672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:06.230675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:06.230864 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:06.231569 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:06.231593 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:06.232041 (+ 448us) client_negotiation.cc:769] Sending connection context
0927 18:45:06.232294 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 18:45:06.232782 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":289,"threads_started":1}
I20260927 18:45:06.235615 3817 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35753
I20260927 18:45:06.235962 3817 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:06.236797 3817 heartbeater.cc:511] Master 127.31.26.254:35753 requested a full tablet report, sending...
I20260927 18:45:06.238600 3496 ts_manager.cc:194] Registered new tserver with Master: 05dd0fc014cc47e5a1a55b70cef4c6ba (127.31.26.194:45521)
I20260927 18:45:06.239946 3496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:46457
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:45:06.272344 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35753
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:06.601152 3823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:06.601517 3823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:06.601693 3823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:06.608986 3823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:06.609186 3823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:06.609391 3823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:06.619532 3823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:06.620887 3823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:06.622354 3823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:06.635718 3831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:06.635819 3828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:06.635828 3829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:06.636533 3823 server_base.cc:1034] running on GCE node
I20260927 18:45:06.637247 3823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:06.638384 3823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:06.639590 3823 hybrid_clock.cc:648] HybridClock initialized: now 1790534706639511 us; error 65 us; skew 500 ppm
I20260927 18:45:06.639943 3823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:06 dist-test-slave-rfd5 krb5kdc[3452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:06.643283 3823 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:45:06.644532 3823 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:06.644659 3823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:06.645030 3823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:06.648241 3823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05"
format_stamp: "Formatted at 2026-09-27 18:45:06 on dist-test-slave-rfd5"
server_key: "d61dd1f117da83c8d5c3564f45728002"
server_key_iv: "17c2d463d31a08e8aa60b55f092f77af"
server_key_version: "encryptionkey@0"
I20260927 18:45:06.649303 3823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05"
format_stamp: "Formatted at 2026-09-27 18:45:06 on dist-test-slave-rfd5"
server_key: "d61dd1f117da83c8d5c3564f45728002"
server_key_iv: "17c2d463d31a08e8aa60b55f092f77af"
server_key_version: "encryptionkey@0"
I20260927 18:45:06.655169 3823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:45:06.658836 3837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:06.660317 3823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:45:06.660521 3823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05"
format_stamp: "Formatted at 2026-09-27 18:45:06 on dist-test-slave-rfd5"
server_key: "d61dd1f117da83c8d5c3564f45728002"
server_key_iv: "17c2d463d31a08e8aa60b55f092f77af"
server_key_version: "encryptionkey@0"
I20260927 18:45:06.660754 3823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:06.691025 3823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:06.695451 3823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:06.695745 3823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:06.696678 3823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:06.698006 3823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:06.698117 3823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:06.698247 3823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:06.698340 3823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:06.718631 3684 heartbeater.cc:503] Master 127.31.26.254:35753 was elected leader, sending a full tablet report...
I20260927 18:45:06.732213 3823 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:45579
I20260927 18:45:06.732384 3949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:45579 every 8 connection(s)
I20260927 18:45:06.733139 3823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:06.741770 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 3823
I20260927 18:45:06.741989 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:06.742414 31851 external_mini_cluster.cc:1428] Setting key fc37fbdb3df0a9e2ffe97c656f58aa28
Sep 27 18:45:06 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:06.754823 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.735787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:53165 (local address 127.31.26.254:35753)
0927 18:45:06.736243 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:06.736250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:06.737491 (+ 1241us) server_negotiation.cc:407] Connection header received
0927 18:45:06.739593 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:06.739604 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:06.739703 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:06.739808 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:06.741761 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.742727 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.744966 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.745636 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.745776 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.749390 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:06.749396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:06.749401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:06.749443 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:06.752099 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:06.752802 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:06.752806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:06.752808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:06.752918 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:06.753345 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:06.753348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:06.753350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:06.753590 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:06.753698 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:06.754402 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 18:45:06.754566 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":201,"threads_started":1}
I20260927 18:45:06.755853 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.736396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35753 (local address 127.31.26.195:53165)
0927 18:45:06.737305 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:06.737365 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:06.739103 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:06.740053 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:06.740063 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:06.740663 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:06.741505 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.741541 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.742897 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.742906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.744813 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.744826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.745874 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.745881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.746117 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.746870 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:06.746899 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:06.749084 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:06.752226 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:06.752233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:06.752247 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:06.752686 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:06.753007 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:06.753010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:06.753013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:06.753257 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:06.753701 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:06.753707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:06.754064 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 18:45:06.754302 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 18:45:06.754701 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":337,"threads_started":1}
I20260927 18:45:06.757521 3950 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35753
I20260927 18:45:06.757882 3950 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:06.758589 3950 heartbeater.cc:511] Master 127.31.26.254:35753 requested a full tablet report, sending...
I20260927 18:45:06.760031 3495 ts_manager.cc:194] Registered new tserver with Master: 3ffe4dcbfd7d4aabbadb2807762d2e05 (127.31.26.195:45579)
I20260927 18:45:06.760934 3495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:53165
I20260927 18:45:06.770792 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:45:06.788199 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.775293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.31.26.254:35753)
0927 18:45:06.775457 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:06.775461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:06.775985 (+ 524us) server_negotiation.cc:407] Connection header received
0927 18:45:06.776107 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:06.776110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:06.776174 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:06.776255 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:06.777838 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.778599 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.779905 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.780317 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.780425 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.782965 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:06.782970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:06.782974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:06.783010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:06.785785 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:06.786496 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:06.786499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:06.786502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:06.786613 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:06.786986 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:06.786989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:06.786991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:06.787295 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:06.787483 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:06.787793 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 18:45:06.787969 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:45:06.792485 3495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:06.799378 3495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:06.821626 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.811350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:45521 (local address 127.0.0.1:33790)
0927 18:45:06.812676 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:06.812725 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:06.812954 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:06.814251 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:06.814258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:06.814288 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:06.814712 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.814731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.816460 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.816469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.819002 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.819015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.820630 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.820636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.820809 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.820859 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 18:45:06.821008 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:45:06.821223 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":1,"thread_start_us":1958,"threads_started":1}
I20260927 18:45:06.822492 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.811483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.31.26.194:45521)
0927 18:45:06.813307 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:06.813322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:06.813352 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:45:06.813454 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:06.813463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:06.813729 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:06.813958 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:06.815030 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.816330 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.819159 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.820447 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.820607 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.820815 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:06.821048 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 18:45:06.821242 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":376,"threads_started":1}
I20260927 18:45:06.822929 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.811428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45579 (local address 127.0.0.1:36322)
0927 18:45:06.813629 (+ 2201us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:06.813650 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:06.813803 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:06.815094 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:06.815099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:06.815128 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:06.815609 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.815621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.817648 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.817655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.819934 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.819947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.822059 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.822069 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.822232 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.822266 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:45:06.822374 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:45:06.822501 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"mutex_wait_us":38,"thread_start_us":804,"threads_started":1}
I20260927 18:45:06.823601 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.814309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:45421 (local address 127.0.0.1:55388)
0927 18:45:06.815275 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:06.815298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:06.815502 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:06.816384 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:06.816388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:06.816411 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:06.816908 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.816919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.818758 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.818762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.820440 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:06.820453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.822830 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.822836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:06.823032 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.823067 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:45:06.823191 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:45:06.823315 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":628,"threads_started":1}
I20260927 18:45:06.823787 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.812070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.31.26.195:45579)
0927 18:45:06.814449 (+ 2379us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:06.814459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:06.814493 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:45:06.814589 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:06.814596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:06.814865 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:06.815786 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:06.816077 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.817515 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.820104 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.821573 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.821699 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.821913 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:06.823265 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0927 18:45:06.823439 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2164,"thread_start_us":2007,"threads_started":1}
I20260927 18:45:06.823948 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:06.814005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.31.26.193:45421)
0927 18:45:06.814707 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:06.814715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:06.815390 (+ 675us) server_negotiation.cc:407] Connection header received
0927 18:45:06.815719 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:06.815728 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:06.815939 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:06.816101 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:06.817105 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.818602 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.820626 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:06.822347 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:06.822445 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:06.822638 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:06.823256 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 18:45:06.823427 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":433,"threads_started":1}
I20260927 18:45:06.826468 3752 tablet_service.cc:1467] Processing CreateTablet for tablet 6534a485317145d2a599f9a315e8b603 (DEFAULT_TABLE table=test-table [id=fab427a28946491589a0fd67a5210c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:06.827288 3885 tablet_service.cc:1467] Processing CreateTablet for tablet 6534a485317145d2a599f9a315e8b603 (DEFAULT_TABLE table=test-table [id=fab427a28946491589a0fd67a5210c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:06.827966 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6534a485317145d2a599f9a315e8b603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:06.827793 3619 tablet_service.cc:1467] Processing CreateTablet for tablet 6534a485317145d2a599f9a315e8b603 (DEFAULT_TABLE table=test-table [id=fab427a28946491589a0fd67a5210c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:06.828704 3885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6534a485317145d2a599f9a315e8b603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:06.829080 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6534a485317145d2a599f9a315e8b603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:06.840981 3969 tablet_bootstrap.cc:492] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Bootstrap starting.
I20260927 18:45:06.843007 3971 tablet_bootstrap.cc:492] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Bootstrap starting.
I20260927 18:45:06.844395 3970 tablet_bootstrap.cc:492] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Bootstrap starting.
I20260927 18:45:06.846017 3969 tablet_bootstrap.cc:654] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:06.847325 3969 log.cc:824] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:06.848047 3970 tablet_bootstrap.cc:654] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:06.849248 3970 log.cc:824] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:06.850673 3969 tablet_bootstrap.cc:492] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: No bootstrap required, opened a new log
I20260927 18:45:06.850950 3969 ts_tablet_manager.cc:1397] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:45:06.851195 3971 tablet_bootstrap.cc:654] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:06.852370 3970 tablet_bootstrap.cc:492] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: No bootstrap required, opened a new log
I20260927 18:45:06.852391 3971 log.cc:824] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:06.852636 3970 ts_tablet_manager.cc:1397] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:45:06.858973 3969 raft_consensus.cc:348] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:06.859361 3969 raft_consensus.cc:374] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:06.859473 3969 raft_consensus.cc:729] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ce013ceebc40f1bcb5de95ec0f5c5c, State: Initialized, Role: FOLLOWER
I20260927 18:45:06.859995 3971 tablet_bootstrap.cc:492] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: No bootstrap required, opened a new log
I20260927 18:45:06.860131 3969 consensus_queue.cc:260] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:06.860272 3971 ts_tablet_manager.cc:1397] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20260927 18:45:06.861888 3969 ts_tablet_manager.cc:1428] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 18:45:06.863976 3970 raft_consensus.cc:348] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:06.864447 3970 raft_consensus.cc:374] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:06.864539 3970 raft_consensus.cc:729] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dd0fc014cc47e5a1a55b70cef4c6ba, State: Initialized, Role: FOLLOWER
I20260927 18:45:06.865391 3970 consensus_queue.cc:260] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:06.866592 3817 heartbeater.cc:503] Master 127.31.26.254:35753 was elected leader, sending a full tablet report...
I20260927 18:45:06.867213 3970 ts_tablet_manager.cc:1428] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 18:45:06.870438 3971 raft_consensus.cc:348] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:06.870818 3971 raft_consensus.cc:374] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:06.870921 3971 raft_consensus.cc:729] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffe4dcbfd7d4aabbadb2807762d2e05, State: Initialized, Role: FOLLOWER
I20260927 18:45:06.871510 3971 consensus_queue.cc:260] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:06.872563 3950 heartbeater.cc:503] Master 127.31.26.254:35753 was elected leader, sending a full tablet report...
I20260927 18:45:06.873255 3971 ts_tablet_manager.cc:1428] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
W20260927 18:45:06.935824 3685 tablet.cc:2369] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:45:06.963960 3818 tablet.cc:2369] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:45:06.984760 3951 tablet.cc:2369] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:07.261325 3975 raft_consensus.cc:482] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:07.261680 3975 raft_consensus.cc:504] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:07.263576 3975 leader_election.cc:288] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05dd0fc014cc47e5a1a55b70cef4c6ba (127.31.26.194:45521), 3ffe4dcbfd7d4aabbadb2807762d2e05 (127.31.26.195:45579)
I20260927 18:45:07.271252 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.264056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:54991 (local address 127.31.26.195:45579)
0927 18:45:07.264246 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:07.264254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:07.265123 (+ 869us) server_negotiation.cc:407] Connection header received
0927 18:45:07.265460 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:07.265464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:07.265523 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:07.265617 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:07.266232 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.266940 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.269283 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.270100 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.270299 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.270381 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:07.270835 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 18:45:07.270970 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:45:07.271293 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.264050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:45277 (local address 127.31.26.194:45521)
0927 18:45:07.264255 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:07.264264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:07.265146 (+ 882us) server_negotiation.cc:407] Connection header received
0927 18:45:07.265389 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:07.265395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:07.265472 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:07.265569 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:07.266186 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.266910 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.269299 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.270105 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.270341 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.270418 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:07.270871 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 18:45:07.271035 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:45:07.271466 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.264037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:45521 (local address 127.31.26.193:45277)
0927 18:45:07.265007 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:07.265022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:07.265232 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:07.265584 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:07.265588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:07.265610 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:07.266014 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:07.266027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.267045 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.267053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:07.269142 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:07.269159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.270376 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.270382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:07.270691 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.270717 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:45:07.270846 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:45:07.270997 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":76,"thread_start_us":496,"threads_started":1}
I20260927 18:45:07.271720 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.264033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45579 (local address 127.31.26.193:54991)
0927 18:45:07.264981 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:07.265000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:07.265263 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:07.265641 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:07.265646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:07.265674 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:07.266082 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:07.266092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.267075 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.267080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:07.269129 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:07.269150 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.270356 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.270363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:07.270654 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.270699 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:45:07.270809 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:45:07.270971 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":74,"spinlock_wait_cycles":1408,"thread_start_us":443,"threads_started":1}
I20260927 18:45:07.272651 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6534a485317145d2a599f9a315e8b603" candidate_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" is_pre_election: true
I20260927 18:45:07.272761 3905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6534a485317145d2a599f9a315e8b603" candidate_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" is_pre_election: true
I20260927 18:45:07.273319 3905 raft_consensus.cc:2454] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ce013ceebc40f1bcb5de95ec0f5c5c in term 0.
I20260927 18:45:07.273319 3772 raft_consensus.cc:2454] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ce013ceebc40f1bcb5de95ec0f5c5c in term 0.
I20260927 18:45:07.274055 3574 leader_election.cc:302] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05dd0fc014cc47e5a1a55b70cef4c6ba, b9ce013ceebc40f1bcb5de95ec0f5c5c; no voters:
I20260927 18:45:07.274521 3975 raft_consensus.cc:2781] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:07.274689 3975 raft_consensus.cc:482] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:07.274792 3975 raft_consensus.cc:3037] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:07.276922 3975 raft_consensus.cc:504] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:07.277633 3975 leader_election.cc:288] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [CANDIDATE]: Term 1 election: Requested vote from peers 05dd0fc014cc47e5a1a55b70cef4c6ba (127.31.26.194:45521), 3ffe4dcbfd7d4aabbadb2807762d2e05 (127.31.26.195:45579)
I20260927 18:45:07.278064 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6534a485317145d2a599f9a315e8b603" candidate_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba"
I20260927 18:45:07.278193 3905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6534a485317145d2a599f9a315e8b603" candidate_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05"
I20260927 18:45:07.278286 3772 raft_consensus.cc:3037] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:07.278405 3905 raft_consensus.cc:3037] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:07.281759 3905 raft_consensus.cc:2454] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ce013ceebc40f1bcb5de95ec0f5c5c in term 1.
I20260927 18:45:07.281778 3772 raft_consensus.cc:2454] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ce013ceebc40f1bcb5de95ec0f5c5c in term 1.
I20260927 18:45:07.282236 3572 leader_election.cc:302] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffe4dcbfd7d4aabbadb2807762d2e05, b9ce013ceebc40f1bcb5de95ec0f5c5c; no voters:
I20260927 18:45:07.282603 3975 raft_consensus.cc:2781] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:07.283080 3975 raft_consensus.cc:686] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 1 LEADER]: Becoming Leader. State: Replica: b9ce013ceebc40f1bcb5de95ec0f5c5c, State: Running, Role: LEADER
I20260927 18:45:07.283654 3975 consensus_queue.cc:237] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } }
I20260927 18:45:07.288604 3495 catalog_manager.cc:5495] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c reported cstate change: term changed from 0 to 1, leader changed from <none> to b9ce013ceebc40f1bcb5de95ec0f5c5c (127.31.26.193). New cstate: current_term: 1 leader_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9ce013ceebc40f1bcb5de95ec0f5c5c" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 45421 } health_report { overall_health: HEALTHY } } }
I20260927 18:45:07.391646 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.385406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.31.26.193:45421)
0927 18:45:07.385862 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:07.385867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:07.386356 (+ 489us) server_negotiation.cc:407] Connection header received
0927 18:45:07.386586 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:07.386590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:07.386681 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:07.386833 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:07.387253 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.387938 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.389499 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.389929 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.390031 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.390568 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:07.391142 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:07.391254 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:07.391357 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 18:45:07.391430 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
I20260927 18:45:07.400043 3905 raft_consensus.cc:1261] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 1 FOLLOWER]: Refusing update from remote peer b9ce013ceebc40f1bcb5de95ec0f5c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:07.400043 3772 raft_consensus.cc:1261] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 1 FOLLOWER]: Refusing update from remote peer b9ce013ceebc40f1bcb5de95ec0f5c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:07.401230 3980 consensus_queue.cc:1035] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffe4dcbfd7d4aabbadb2807762d2e05" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:07.401649 3975 consensus_queue.cc:1035] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05dd0fc014cc47e5a1a55b70cef4c6ba" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:07.410986 3985 mvcc.cc:205] Tried to move back new op lower bound from 7334030161499648000 to 7334030161037680640. Current Snapshot: MvccSnapshot[applied={T|T < 7334030161499648000}]
I20260927 18:45:07.416290 3989 mvcc.cc:205] Tried to move back new op lower bound from 7334030161499648000 to 7334030161037680640. Current Snapshot: MvccSnapshot[applied={T|T < 7334030161499648000 or (T in {7334030161499648000})}]
I20260927 18:45:07.429612 3495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:07.429857 3495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:07.434126 3495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d552ec3dce3743a59de23a53f02b89e9: Sending DeleteTablet for 3 replicas of tablet 6534a485317145d2a599f9a315e8b603
I20260927 18:45:07.435570 3752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6534a485317145d2a599f9a315e8b603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:07 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:33790
I20260927 18:45:07.435622 3885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6534a485317145d2a599f9a315e8b603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:07 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:36322
I20260927 18:45:07.436350 3619 tablet_service.cc:1514] Processing DeleteTablet for tablet 6534a485317145d2a599f9a315e8b603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:07 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:55388
I20260927 18:45:07.436826 3998 tablet_replica.cc:330] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: stopping tablet replica
I20260927 18:45:07.437508 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3557
I20260927 18:45:07.437469 3998 raft_consensus.cc:2229] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:07.438221 3998 raft_consensus.cc:2258] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:07.439129 3999 tablet_replica.cc:330] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: stopping tablet replica
I20260927 18:45:07.439559 3999 raft_consensus.cc:2229] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:07.440183 3999 raft_consensus.cc:2258] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:07.440521 3998 ts_tablet_manager.cc:1905] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:07.441977 3999 ts_tablet_manager.cc:1905] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:07.447821 3998 ts_tablet_manager.cc:1918] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:07.448036 3998 log.cc:1196] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/6534a485317145d2a599f9a315e8b603
I20260927 18:45:07.448553 3998 ts_tablet_manager.cc:1939] T 6534a485317145d2a599f9a315e8b603 P 3ffe4dcbfd7d4aabbadb2807762d2e05: Deleting consensus metadata
I20260927 18:45:07.449347 3999 ts_tablet_manager.cc:1918] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:07.449539 3999 log.cc:1196] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/6534a485317145d2a599f9a315e8b603
I20260927 18:45:07.449674 3997 tablet_replica.cc:330] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: stopping tablet replica
I20260927 18:45:07.450171 3999 ts_tablet_manager.cc:1939] T 6534a485317145d2a599f9a315e8b603 P b9ce013ceebc40f1bcb5de95ec0f5c5c: Deleting consensus metadata
I20260927 18:45:07.450284 3997 raft_consensus.cc:2229] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:07.450788 3480 catalog_manager.cc:4841] TS 3ffe4dcbfd7d4aabbadb2807762d2e05 (127.31.26.195:45579): tablet 6534a485317145d2a599f9a315e8b603 (table test-table [id=fab427a28946491589a0fd67a5210c4d]) successfully deleted
I20260927 18:45:07.450937 3997 raft_consensus.cc:2258] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:07.451858 3482 catalog_manager.cc:4841] TS b9ce013ceebc40f1bcb5de95ec0f5c5c (127.31.26.193:45421): tablet 6534a485317145d2a599f9a315e8b603 (table test-table [id=fab427a28946491589a0fd67a5210c4d]) successfully deleted
I20260927 18:45:07.453540 3997 ts_tablet_manager.cc:1905] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:07.461156 3997 ts_tablet_manager.cc:1918] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:07.461383 3997 log.cc:1196] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/6534a485317145d2a599f9a315e8b603
Sep 27 18:45:07 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:45:07.461923 3997 ts_tablet_manager.cc:1939] T 6534a485317145d2a599f9a315e8b603 P 05dd0fc014cc47e5a1a55b70cef4c6ba: Deleting consensus metadata
I20260927 18:45:07.463867 3482 catalog_manager.cc:4841] TS 05dd0fc014cc47e5a1a55b70cef4c6ba (127.31.26.194:45521): tablet 6534a485317145d2a599f9a315e8b603 (table test-table [id=fab427a28946491589a0fd67a5210c4d]) successfully deleted
I20260927 18:45:07.470310 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.439937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.31.26.193:45421)
0927 18:45:07.446904 (+ 6967us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:07.446918 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:07.453699 (+ 6781us) server_negotiation.cc:407] Connection header received
0927 18:45:07.453993 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:07.453999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:07.454080 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:07.454192 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:07.455489 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.456324 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.457681 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.458140 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.458318 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.462728 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:07.462743 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:07.462760 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:07.462799 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:07.466954 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:07.467794 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:07.467805 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:07.467809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:07.467994 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:07.468414 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:07.468420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:07.468423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:07.468920 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:07.469289 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:07.469608 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 18:45:07.469918 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6750}
I20260927 18:45:07.471101 3679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:07.570909 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3557
I20260927 18:45:07.590229 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3690
Sep 27 18:45:07 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:45:07.605432 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.591035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.31.26.194:45521)
0927 18:45:07.591201 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:07.591207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:07.591229 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:45:07.591416 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:07.591420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:07.591490 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:07.591576 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:07.592903 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.593777 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.595071 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.595468 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.595570 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.599813 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:07.599829 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:07.599846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:07.599905 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:07.602890 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:07.603596 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:07.603603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:07.603606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:07.603742 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:07.604069 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:07.604072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:07.604074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:07.604437 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:07.604652 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:07.604921 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 18:45:07.605155 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:45:07.606088 3812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:07.695365 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3690
I20260927 18:45:07.712891 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3823
Sep 27 18:45:07 dist-test-slave-rfd5 krb5kdc[3452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:45:07.730583 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:07.713899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.31.26.195:45579)
0927 18:45:07.714112 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:07.714116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:07.714137 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:45:07.714367 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:07.714372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:07.714440 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:07.714536 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:07.716015 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.716824 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.718209 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:07.718680 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:07.718794 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:07.723719 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:07.723737 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:07.723755 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:07.723811 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:07.727159 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:07.728077 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:07.728086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:07.728088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:07.728312 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:07.728787 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:07.728793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:07.728796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:07.729358 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:07.729622 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:07.729968 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 18:45:07.730148 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:45:07.731271 3945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:07.822438 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3823
I20260927 18:45:07.839512 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 3465
I20260927 18:45:07.840777 3526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:07.949820 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 3465
2026-09-27T18:45:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4647 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T18:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:08Z Disabled control of system clock
I20260927 18:45:08.024406 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44403
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:45:08.380105 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:08.380512 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:08.380678 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:08.388332 4008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:08.388520 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:08.388613 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:08.388721 4008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:08.388808 4008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:08.398725 4008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:08.400167 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:08.402092 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:08.414635 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:08.414665 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:08.414561 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:08.415608 4008 server_base.cc:1034] running on GCE node
I20260927 18:45:08.416501 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:08.418187 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:08.419494 4008 hybrid_clock.cc:648] HybridClock initialized: now 1790534708419459 us; error 41 us; skew 500 ppm
I20260927 18:45:08.419911 4008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:08.422526 4008 webserver.cc:460] Webserver started at http://127.31.26.254:38641/ using document root <none> and password file <none>
I20260927 18:45:08.423277 4008 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:08.423416 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:08.423751 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:08.426898 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "0e4c26ac8bff43d5b494046d1500ec39"
format_stamp: "Formatted at 2026-09-27 18:45:08 on dist-test-slave-rfd5"
server_key: "00d8d0f8202bc32bbdea935e781bd5bc"
server_key_iv: "6dcc907ad9112745bb67035e0fc7023b"
server_key_version: "encryptionkey@0"
I20260927 18:45:08.427762 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "0e4c26ac8bff43d5b494046d1500ec39"
format_stamp: "Formatted at 2026-09-27 18:45:08 on dist-test-slave-rfd5"
server_key: "00d8d0f8202bc32bbdea935e781bd5bc"
server_key_iv: "6dcc907ad9112745bb67035e0fc7023b"
server_key_version: "encryptionkey@0"
I20260927 18:45:08.433225 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 18:45:08.437039 4022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:08.438582 4008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:08.438782 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "0e4c26ac8bff43d5b494046d1500ec39"
format_stamp: "Formatted at 2026-09-27 18:45:08 on dist-test-slave-rfd5"
server_key: "00d8d0f8202bc32bbdea935e781bd5bc"
server_key_iv: "6dcc907ad9112745bb67035e0fc7023b"
server_key_version: "encryptionkey@0"
I20260927 18:45:08.439024 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:08.445009 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:08.445942 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:08.446259 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:08.465348 4008 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44403
I20260927 18:45:08.465389 4073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44403 every 8 connection(s)
I20260927 18:45:08.466775 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:08.470429 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4008
I20260927 18:45:08.470664 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:08.471083 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:08.471536 31851 external_mini_cluster.cc:1428] Setting key 2af2fad20a01e90197c0b9745231ff96
I20260927 18:45:08.480785 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:08.473806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.31.26.254:44403)
0927 18:45:08.475005 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:08.475020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:08.475071 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:45:08.476624 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:08.476650 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:08.477301 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:08.477771 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:08.478982 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:08.478989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:08.479002 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:08.479018 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:08.479083 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:08.479099 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:08.479377 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:08.479539 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 18:45:08.479814 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":448,"threads_started":1}
I20260927 18:45:08.485248 4074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39: Bootstrap starting.
I20260927 18:45:08.488692 4074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:08.489965 4074 log.cc:824] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:08.492906 4074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39: No bootstrap required, opened a new log
I20260927 18:45:08.501338 4074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c26ac8bff43d5b494046d1500ec39" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44403 } }
I20260927 18:45:08.501726 4074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:08.501861 4074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4c26ac8bff43d5b494046d1500ec39, State: Initialized, Role: FOLLOWER
I20260927 18:45:08.502540 4074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c26ac8bff43d5b494046d1500ec39" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44403 } }
I20260927 18:45:08.502789 4074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:08.502924 4074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:08.503096 4074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:08.505381 4074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c26ac8bff43d5b494046d1500ec39" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44403 } }
I20260927 18:45:08.505945 4074 leader_election.cc:302] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e4c26ac8bff43d5b494046d1500ec39; no voters:
I20260927 18:45:08.506599 4074 leader_election.cc:288] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:08.506793 4079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:08.507532 4079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [term 1 LEADER]: Becoming Leader. State: Replica: 0e4c26ac8bff43d5b494046d1500ec39, State: Running, Role: LEADER
I20260927 18:45:08.508096 4079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c26ac8bff43d5b494046d1500ec39" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44403 } }
I20260927 18:45:08.509014 4074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:08.511677 4081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e4c26ac8bff43d5b494046d1500ec39. Latest consensus state: current_term: 1 leader_uuid: "0e4c26ac8bff43d5b494046d1500ec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c26ac8bff43d5b494046d1500ec39" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44403 } } }
I20260927 18:45:08.512118 4081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:08.511708 4080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e4c26ac8bff43d5b494046d1500ec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c26ac8bff43d5b494046d1500ec39" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44403 } } }
I20260927 18:45:08.512262 4080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:08.518512 4085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:08.523041 4085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:08.532078 4085 catalog_manager.cc:1347] Generated new cluster ID: 91214c9e78f94ea4927f62018d8f5922
I20260927 18:45:08.532359 4085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:08.547258 4085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:08.547717 4085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:08.558758 4085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39: Generated new TSK 0
I20260927 18:45:08.559718 4085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:45:08.571517 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44403
--builtin_ntp_servers=127.31.26.212:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:45:08.907773 4098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:08.908128 4098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:08.908294 4098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:08.915998 4098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:08.916147 4098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:08.916328 4098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:08.925598 4098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:08.926936 4098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:08.928782 4098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:08.941987 4103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:08.942377 4106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:08.942484 4104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:08.943094 4098 server_base.cc:1034] running on GCE node
I20260927 18:45:08.943794 4098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:08.944726 4098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:08.945935 4098 hybrid_clock.cc:648] HybridClock initialized: now 1790534708945890 us; error 38 us; skew 500 ppm
I20260927 18:45:08.946278 4098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:08.948666 4098 webserver.cc:460] Webserver started at http://127.31.26.193:43147/ using document root <none> and password file <none>
I20260927 18:45:08.949463 4098 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:08.949604 4098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:08.949960 4098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:08.952941 4098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2"
format_stamp: "Formatted at 2026-09-27 18:45:08 on dist-test-slave-rfd5"
server_key: "e72ae40fb93a1d028cd5057a502ba6a3"
server_key_iv: "a68d48456d4102813894033c98f431ef"
server_key_version: "encryptionkey@0"
I20260927 18:45:08.953881 4098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2"
format_stamp: "Formatted at 2026-09-27 18:45:08 on dist-test-slave-rfd5"
server_key: "e72ae40fb93a1d028cd5057a502ba6a3"
server_key_iv: "a68d48456d4102813894033c98f431ef"
server_key_version: "encryptionkey@0"
I20260927 18:45:08.959326 4098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:45:08.963291 4112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:08.964987 4098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:08.965193 4098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2"
format_stamp: "Formatted at 2026-09-27 18:45:08 on dist-test-slave-rfd5"
server_key: "e72ae40fb93a1d028cd5057a502ba6a3"
server_key_iv: "a68d48456d4102813894033c98f431ef"
server_key_version: "encryptionkey@0"
I20260927 18:45:08.965395 4098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:08.971095 4098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:08.971879 4098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:08.972103 4098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:08.973035 4098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:08.974496 4098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:08.974581 4098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:08.974661 4098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:08.974725 4098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.008162 4098 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:37367
I20260927 18:45:09.008213 4224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:37367 every 8 connection(s)
I20260927 18:45:09.009761 4098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:09.013427 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4098
I20260927 18:45:09.013630 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:09.014357 31851 external_mini_cluster.cc:1428] Setting key cd00ce2593103728a6ff2f507a018c89
I20260927 18:45:09.017629 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44403
--builtin_ntp_servers=127.31.26.212:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:45:09.019418 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.012433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:46393 (local address 127.31.26.254:44403)
0927 18:45:09.012876 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.012881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.014493 (+ 1612us) server_negotiation.cc:407] Connection header received
0927 18:45:09.016634 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.016640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.016742 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.016872 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.018367 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.018377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.018386 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.018413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.018476 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.018498 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.018641 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.018875 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.018949 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":257,"threads_started":1}
I20260927 18:45:09.020254 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.013019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44403 (local address 127.31.26.193:46393)
0927 18:45:09.014253 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:09.014305 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:09.016303 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:09.017166 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:09.017175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:09.017278 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:09.018051 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:09.018053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:09.018093 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:09.018180 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:09.018635 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:09.018644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:09.018651 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:45:09.018912 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 18:45:09.019204 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":345,"threads_started":1}
I20260927 18:45:09.021927 4225 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44403
I20260927 18:45:09.022250 4225 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:09.022857 4225 heartbeater.cc:511] Master 127.31.26.254:44403 requested a full tablet report, sending...
I20260927 18:45:09.025022 4039 ts_manager.cc:194] Registered new tserver with Master: fdd8bacf1ee84dd1ac25ddfafb027eb2 (127.31.26.193:37367)
W20260927 18:45:09.353533 4229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:09.353842 4229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:09.354018 4229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:09.361371 4229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:09.361478 4229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:09.361667 4229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:09.374811 4229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:09.376163 4229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:09.378011 4229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:09.390973 4234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:09.391000 4237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:09.390951 4235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:09.391981 4229 server_base.cc:1034] running on GCE node
I20260927 18:45:09.392628 4229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:09.393756 4229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:09.395001 4229 hybrid_clock.cc:648] HybridClock initialized: now 1790534709394963 us; error 52 us; skew 500 ppm
I20260927 18:45:09.395366 4229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:09.397908 4229 webserver.cc:460] Webserver started at http://127.31.26.194:35727/ using document root <none> and password file <none>
I20260927 18:45:09.398640 4229 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:09.398774 4229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:09.399111 4229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:09.402149 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "f07a8d574448488198e6b3976222fd90"
format_stamp: "Formatted at 2026-09-27 18:45:09 on dist-test-slave-rfd5"
server_key: "9c05a6a60904c225ba1e47079506ad5d"
server_key_iv: "0cf85b1131006575e7d9ecf0ea6dfa13"
server_key_version: "encryptionkey@0"
I20260927 18:45:09.403183 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "f07a8d574448488198e6b3976222fd90"
format_stamp: "Formatted at 2026-09-27 18:45:09 on dist-test-slave-rfd5"
server_key: "9c05a6a60904c225ba1e47079506ad5d"
server_key_iv: "0cf85b1131006575e7d9ecf0ea6dfa13"
server_key_version: "encryptionkey@0"
I20260927 18:45:09.408773 4229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:09.412665 4243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.414306 4229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:09.414513 4229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "f07a8d574448488198e6b3976222fd90"
format_stamp: "Formatted at 2026-09-27 18:45:09 on dist-test-slave-rfd5"
server_key: "9c05a6a60904c225ba1e47079506ad5d"
server_key_iv: "0cf85b1131006575e7d9ecf0ea6dfa13"
server_key_version: "encryptionkey@0"
I20260927 18:45:09.414760 4229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:09.420490 4229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:09.421401 4229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:09.421681 4229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:09.422619 4229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:09.424108 4229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:09.424263 4229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.424397 4229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:09.424494 4229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.457269 4229 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:44351
I20260927 18:45:09.457316 4355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:44351 every 8 connection(s)
I20260927 18:45:09.458624 4229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:09.468043 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.461355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:46361 (local address 127.31.26.254:44403)
0927 18:45:09.461595 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.461603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.463379 (+ 1776us) server_negotiation.cc:407] Connection header received
0927 18:45:09.465577 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.465582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.465635 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.465731 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.467333 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.467337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.467340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.467356 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.467386 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.467400 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.467498 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.467724 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.467775 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:45:09.468462 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4229
I20260927 18:45:09.468676 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:09.469240 31851 external_mini_cluster.cc:1428] Setting key b62f8c8c232ee80f90346d2dbf2c8777
I20260927 18:45:09.469270 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.462027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44403 (local address 127.31.26.194:46361)
0927 18:45:09.463183 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:09.463237 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:09.465253 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:09.466034 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:09.466048 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:09.466196 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:09.466953 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:09.466957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:09.467011 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:09.467141 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:09.467505 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:09.467515 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:09.467523 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:45:09.467791 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 18:45:09.468041 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":382,"threads_started":1}
I20260927 18:45:09.471232 4356 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44403
I20260927 18:45:09.471591 4356 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:09.472293 4356 heartbeater.cc:511] Master 127.31.26.254:44403 requested a full tablet report, sending...
I20260927 18:45:09.472625 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44403
--builtin_ntp_servers=127.31.26.212:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:45:09.473954 4039 ts_manager.cc:194] Registered new tserver with Master: f07a8d574448488198e6b3976222fd90 (127.31.26.194:44351)
W20260927 18:45:09.803565 4359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:09.803921 4359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:09.804087 4359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:09.811617 4359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:09.811766 4359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:09.811946 4359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:09.821491 4359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:09.822883 4359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:09.824712 4359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:09.838651 4365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:09.838565 4367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:09.838528 4364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:09.839835 4359 server_base.cc:1034] running on GCE node
I20260927 18:45:09.840461 4359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:09.841598 4359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:09.842849 4359 hybrid_clock.cc:648] HybridClock initialized: now 1790534709842794 us; error 46 us; skew 500 ppm
I20260927 18:45:09.843206 4359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:09.845727 4359 webserver.cc:460] Webserver started at http://127.31.26.195:33881/ using document root <none> and password file <none>
I20260927 18:45:09.846473 4359 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:09.846608 4359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:09.846966 4359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:09.850049 4359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "9a24c41492b24e6d89e607500338fc81"
format_stamp: "Formatted at 2026-09-27 18:45:09 on dist-test-slave-rfd5"
server_key: "cf91c5926f83e15636847bcca5df0fb1"
server_key_iv: "aaa3c0a252a4585c537942afab5b581d"
server_key_version: "encryptionkey@0"
I20260927 18:45:09.850909 4359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "9a24c41492b24e6d89e607500338fc81"
format_stamp: "Formatted at 2026-09-27 18:45:09 on dist-test-slave-rfd5"
server_key: "cf91c5926f83e15636847bcca5df0fb1"
server_key_iv: "aaa3c0a252a4585c537942afab5b581d"
server_key_version: "encryptionkey@0"
I20260927 18:45:09.856472 4359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:09.860471 4373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.862396 4359 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:45:09.862604 4359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "9a24c41492b24e6d89e607500338fc81"
format_stamp: "Formatted at 2026-09-27 18:45:09 on dist-test-slave-rfd5"
server_key: "cf91c5926f83e15636847bcca5df0fb1"
server_key_iv: "aaa3c0a252a4585c537942afab5b581d"
server_key_version: "encryptionkey@0"
I20260927 18:45:09.862888 4359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:09.869004 4359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:09.870127 4359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:09.870739 4359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:09.871796 4359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:09.873441 4359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:09.873570 4359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.873703 4359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:09.873790 4359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:09.907573 4359 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:34751
I20260927 18:45:09.907665 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:34751 every 8 connection(s)
I20260927 18:45:09.908988 4359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:09.914395 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4359
I20260927 18:45:09.914580 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:09.915141 31851 external_mini_cluster.cc:1428] Setting key e5bbefb845a9cb7c1cae51e68ff5259b
I20260927 18:45:09.918594 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.911664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:55019 (local address 127.31.26.254:44403)
0927 18:45:09.911877 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.911883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.913385 (+ 1502us) server_negotiation.cc:407] Connection header received
0927 18:45:09.915782 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.915789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.915875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.916000 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.917806 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.917812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.917817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.917838 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.917876 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.917893 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.918028 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.918258 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.918319 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:45:09.919801 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.912250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44403 (local address 127.31.26.195:55019)
0927 18:45:09.913216 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:09.913252 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:09.915387 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:09.916279 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:09.916288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:09.916485 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:09.917486 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:09.917490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:09.917542 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:09.917634 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:09.918009 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:09.918017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:09.918024 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:45:09.918337 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 18:45:09.918627 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260927 18:45:09.921066 4486 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44403
I20260927 18:45:09.921378 4486 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:09.921942 4486 heartbeater.cc:511] Master 127.31.26.254:44403 requested a full tablet report, sending...
I20260927 18:45:09.923192 4039 ts_manager.cc:194] Registered new tserver with Master: 9a24c41492b24e6d89e607500338fc81 (127.31.26.195:34751)
I20260927 18:45:09.930817 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:45:09.937773 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.935310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.31.26.254:44403)
0927 18:45:09.935471 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.935476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.935990 (+ 514us) server_negotiation.cc:407] Connection header received
0927 18:45:09.936121 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.936125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.936177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.936254 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.937194 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.937198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.937201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.937216 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.937244 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.937255 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.937382 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.937523 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.937567 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:45:09.941802 4039 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:09.948776 4039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:09.969000 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.962143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.31.26.194:44351)
0927 18:45:09.965306 (+ 3163us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.965318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.965351 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:45:09.965425 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.965436 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.965717 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.966627 (+ 910us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.968168 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.968174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.968184 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.968202 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.968248 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.968262 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.968443 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.968516 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.968619 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":2843,"threads_started":1}
I20260927 18:45:09.969046 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.962661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.31.26.195:34751)
0927 18:45:09.964837 (+ 2176us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.964849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.965700 (+ 851us) server_negotiation.cc:407] Connection header received
0927 18:45:09.965788 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.965798 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.966002 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.966147 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.967427 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.967436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.967451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.967467 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.967525 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.967550 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.967783 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.968570 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.968662 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":1796,"threads_started":1}
I20260927 18:45:09.969493 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.961877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:34751 (local address 127.0.0.1:37766)
0927 18:45:09.965462 (+ 3585us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:09.965485 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:09.965585 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:09.966506 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:09.966514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:09.966543 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:09.967208 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:09.967214 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:09.967231 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:09.967276 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:09.967738 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:09.967752 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:09.967761 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:45:09.967930 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:45:09.968025 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3445,"thread_start_us":1583,"threads_started":1}
I20260927 18:45:09.969889 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.961650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:44351 (local address 127.0.0.1:59822)
0927 18:45:09.964110 (+ 2460us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:09.964170 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:09.964334 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:09.965941 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:09.965948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:09.966046 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:09.967022 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:09.967026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:09.967056 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:09.967145 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:09.968372 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:09.968377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:09.968382 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:45:09.968572 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 18:45:09.968618 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2090,"thread_start_us":2927,"threads_started":1}
I20260927 18:45:09.971601 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.965250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.31.26.193:37367)
0927 18:45:09.966377 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:09.966385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:09.967675 (+ 1290us) server_negotiation.cc:407] Connection header received
0927 18:45:09.967817 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:09.967828 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:09.968090 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:09.968213 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:09.970826 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:09.970838 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:09.970853 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:09.970872 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:09.970937 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:09.970956 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:09.971111 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:09.971183 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:45:09.971288 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":962,"threads_started":1}
I20260927 18:45:09.971704 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:09.966044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:42948)
0927 18:45:09.967550 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:09.967566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:09.967680 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:09.969577 (+ 1897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:09.969584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:09.969607 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:09.970600 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:09.970604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:09.970620 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:09.970662 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:09.971068 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:09.971073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:09.971078 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:45:09.971206 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 18:45:09.971264 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"thread_start_us":1003,"threads_started":1}
I20260927 18:45:09.973865 4291 tablet_service.cc:1467] Processing CreateTablet for tablet bda93709eb444ab29af65b43c7659e77 (DEFAULT_TABLE table=test-table [id=82835284f10541159e8f4a2a6f0fcbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:09.975194 4291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda93709eb444ab29af65b43c7659e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:09.974675 4421 tablet_service.cc:1467] Processing CreateTablet for tablet bda93709eb444ab29af65b43c7659e77 (DEFAULT_TABLE table=test-table [id=82835284f10541159e8f4a2a6f0fcbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:09.975332 4160 tablet_service.cc:1467] Processing CreateTablet for tablet bda93709eb444ab29af65b43c7659e77 (DEFAULT_TABLE table=test-table [id=82835284f10541159e8f4a2a6f0fcbcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:09.976106 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda93709eb444ab29af65b43c7659e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:09.976588 4160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda93709eb444ab29af65b43c7659e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:09.987403 4504 tablet_bootstrap.cc:492] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Bootstrap starting.
I20260927 18:45:09.990857 4504 tablet_bootstrap.cc:654] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:09.991153 4505 tablet_bootstrap.cc:492] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: Bootstrap starting.
I20260927 18:45:09.993521 4504 log.cc:824] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:09.995467 4505 tablet_bootstrap.cc:654] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:09.996866 4505 log.cc:824] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:09.996974 4504 tablet_bootstrap.cc:492] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: No bootstrap required, opened a new log
I20260927 18:45:09.997226 4506 tablet_bootstrap.cc:492] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Bootstrap starting.
I20260927 18:45:09.997290 4504 ts_tablet_manager.cc:1397] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260927 18:45:10.000141 4505 tablet_bootstrap.cc:492] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: No bootstrap required, opened a new log
I20260927 18:45:10.000425 4505 ts_tablet_manager.cc:1397] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 18:45:10.000442 4506 tablet_bootstrap.cc:654] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:10.001688 4506 log.cc:824] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:10.004483 4506 tablet_bootstrap.cc:492] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: No bootstrap required, opened a new log
I20260927 18:45:10.004769 4506 ts_tablet_manager.cc:1397] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 18:45:10.009192 4504 raft_consensus.cc:348] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.009711 4504 raft_consensus.cc:374] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:10.009847 4504 raft_consensus.cc:729] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07a8d574448488198e6b3976222fd90, State: Initialized, Role: FOLLOWER
I20260927 18:45:10.010617 4504 consensus_queue.cc:260] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.011452 4505 raft_consensus.cc:348] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.012084 4505 raft_consensus.cc:374] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:10.012260 4505 raft_consensus.cc:729] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd8bacf1ee84dd1ac25ddfafb027eb2, State: Initialized, Role: FOLLOWER
I20260927 18:45:10.013253 4505 consensus_queue.cc:260] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.014259 4506 raft_consensus.cc:348] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.014793 4506 raft_consensus.cc:374] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:10.014988 4506 raft_consensus.cc:729] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a24c41492b24e6d89e607500338fc81, State: Initialized, Role: FOLLOWER
I20260927 18:45:10.015756 4506 consensus_queue.cc:260] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.018431 4356 heartbeater.cc:503] Master 127.31.26.254:44403 was elected leader, sending a full tablet report...
I20260927 18:45:10.019531 4504 ts_tablet_manager.cc:1428] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Time spent starting tablet: real 0.022s user 0.011s sys 0.007s
I20260927 18:45:10.020649 4506 ts_tablet_manager.cc:1428] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 18:45:10.021188 4486 heartbeater.cc:503] Master 127.31.26.254:44403 was elected leader, sending a full tablet report...
I20260927 18:45:10.024385 4225 heartbeater.cc:503] Master 127.31.26.254:44403 was elected leader, sending a full tablet report...
I20260927 18:45:10.025162 4505 ts_tablet_manager.cc:1428] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: Time spent starting tablet: real 0.025s user 0.020s sys 0.004s
I20260927 18:45:10.122483 4511 raft_consensus.cc:482] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:10.122812 4511 raft_consensus.cc:504] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.124598 4511 leader_election.cc:288] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07a8d574448488198e6b3976222fd90 (127.31.26.194:44351), 9a24c41492b24e6d89e607500338fc81 (127.31.26.195:34751)
I20260927 18:45:10.128167 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.125160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:34751 (local address 127.31.26.193:60641)
0927 18:45:10.125933 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.125956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:10.126091 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:10.126467 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:10.126470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:10.126489 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:10.127260 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:10.127262 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:10.127276 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:10.127313 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:10.127678 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:10.127681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:10.127685 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:45:10.127785 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:45:10.127826 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":415,"threads_started":1}
I20260927 18:45:10.128208 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.125136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:60641 (local address 127.31.26.195:34751)
0927 18:45:10.125376 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:10.125384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:10.126059 (+ 675us) server_negotiation.cc:407] Connection header received
0927 18:45:10.126213 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:10.126220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:10.126306 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:10.126452 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:10.127465 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:10.127474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:10.127481 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:10.127506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:10.127555 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:10.127576 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:10.127736 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:10.127791 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:45:10.127858 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:45:10.129468 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda93709eb444ab29af65b43c7659e77" candidate_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a24c41492b24e6d89e607500338fc81" is_pre_election: true
I20260927 18:45:10.130102 4441 raft_consensus.cc:2454] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd8bacf1ee84dd1ac25ddfafb027eb2 in term 0.
I20260927 18:45:10.130730 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.125791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:52263 (local address 127.31.26.194:44351)
0927 18:45:10.125971 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:10.125979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:10.128490 (+ 2511us) server_negotiation.cc:407] Connection header received
0927 18:45:10.128716 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:10.128726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:10.128805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:10.128932 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:10.129967 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:10.129972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:10.129976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:10.129995 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:10.130028 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:10.130045 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:10.130144 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:10.130372 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 18:45:10.130439 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:45:10.130960 4115 leader_election.cc:302] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a24c41492b24e6d89e607500338fc81, fdd8bacf1ee84dd1ac25ddfafb027eb2; no voters:
I20260927 18:45:10.131287 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.125815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:44351 (local address 127.31.26.193:52263)
0927 18:45:10.128409 (+ 2594us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.128426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:10.128561 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:10.128978 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:10.128986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:10.129031 (+ 45us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:10.129714 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:45:10.129717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:10.129735 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:45:10.129782 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:10.130237 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:10.130244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:10.130248 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:45:10.130925 (+ 677us) client_negotiation.cc:240] Negotiation successful
0927 18:45:10.130984 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2495,"thread_start_us":4815,"threads_started":1}
I20260927 18:45:10.131384 4511 raft_consensus.cc:2781] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:10.131554 4511 raft_consensus.cc:482] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:10.131668 4511 raft_consensus.cc:3037] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:10.132135 4311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda93709eb444ab29af65b43c7659e77" candidate_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07a8d574448488198e6b3976222fd90" is_pre_election: true
I20260927 18:45:10.132613 4311 raft_consensus.cc:2454] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdd8bacf1ee84dd1ac25ddfafb027eb2 in term 0.
I20260927 18:45:10.134405 4511 raft_consensus.cc:504] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.134989 4511 leader_election.cc:288] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f07a8d574448488198e6b3976222fd90 (127.31.26.194:44351), 9a24c41492b24e6d89e607500338fc81 (127.31.26.195:34751)
I20260927 18:45:10.135473 4311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda93709eb444ab29af65b43c7659e77" candidate_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07a8d574448488198e6b3976222fd90"
I20260927 18:45:10.135528 4441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda93709eb444ab29af65b43c7659e77" candidate_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a24c41492b24e6d89e607500338fc81"
I20260927 18:45:10.135708 4441 raft_consensus.cc:3037] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:10.135711 4311 raft_consensus.cc:3037] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:10.139076 4441 raft_consensus.cc:2454] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd8bacf1ee84dd1ac25ddfafb027eb2 in term 1.
I20260927 18:45:10.139076 4311 raft_consensus.cc:2454] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdd8bacf1ee84dd1ac25ddfafb027eb2 in term 1.
I20260927 18:45:10.139521 4115 leader_election.cc:302] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a24c41492b24e6d89e607500338fc81, fdd8bacf1ee84dd1ac25ddfafb027eb2; no voters:
I20260927 18:45:10.139848 4511 raft_consensus.cc:2781] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:10.140319 4511 raft_consensus.cc:686] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 1 LEADER]: Becoming Leader. State: Replica: fdd8bacf1ee84dd1ac25ddfafb027eb2, State: Running, Role: LEADER
I20260927 18:45:10.140786 4511 consensus_queue.cc:237] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } }
I20260927 18:45:10.145372 4038 catalog_manager.cc:5495] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdd8bacf1ee84dd1ac25ddfafb027eb2 (127.31.26.193). New cstate: current_term: 1 leader_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd8bacf1ee84dd1ac25ddfafb027eb2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37367 } health_report { overall_health: HEALTHY } } }
W20260927 18:45:10.160810 4487 tablet.cc:2369] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:10.162691 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.160387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.31.26.193:37367)
0927 18:45:10.160608 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:10.160613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:10.160669 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:45:10.160891 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:10.160895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:10.161004 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:10.161171 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:10.162070 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:10.162076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:10.162081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:10.162099 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:10.162137 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:10.162148 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:10.162265 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:10.162365 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:45:10.162428 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:45:10.171147 4311 raft_consensus.cc:1261] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 1 FOLLOWER]: Refusing update from remote peer fdd8bacf1ee84dd1ac25ddfafb027eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:10.171147 4441 raft_consensus.cc:1261] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 1 FOLLOWER]: Refusing update from remote peer fdd8bacf1ee84dd1ac25ddfafb027eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:10.172233 4515 consensus_queue.cc:1035] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07a8d574448488198e6b3976222fd90" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:10.172580 4511 consensus_queue.cc:1035] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a24c41492b24e6d89e607500338fc81" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:10.186890 4519 mvcc.cc:205] Tried to move back new op lower bound from 7334030172850151424 to 7334030172740681728. Current Snapshot: MvccSnapshot[applied={T|T < 7334030172850151424}]
I20260927 18:45:10.187781 4520 mvcc.cc:205] Tried to move back new op lower bound from 7334030172850151424 to 7334030172740681728. Current Snapshot: MvccSnapshot[applied={T|T < 7334030172850151424}]
I20260927 18:45:10.196959 4038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:10.197273 4038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:10.200759 4038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e4c26ac8bff43d5b494046d1500ec39: Sending DeleteTablet for 3 replicas of tablet bda93709eb444ab29af65b43c7659e77
I20260927 18:45:10.201894 4291 tablet_service.cc:1514] Processing DeleteTablet for tablet bda93709eb444ab29af65b43c7659e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:10 UTC) from {username='slave'} at 127.0.0.1:59822
I20260927 18:45:10.201958 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet bda93709eb444ab29af65b43c7659e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:10 UTC) from {username='slave'} at 127.0.0.1:37766
I20260927 18:45:10.201996 4160 tablet_service.cc:1514] Processing DeleteTablet for tablet bda93709eb444ab29af65b43c7659e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:10 UTC) from {username='slave'} at 127.0.0.1:42948
I20260927 18:45:10.205567 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4098
I20260927 18:45:10.214417 4528 tablet_replica.cc:330] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: stopping tablet replica
I20260927 18:45:10.215065 4528 raft_consensus.cc:2229] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:10.215826 4527 tablet_replica.cc:330] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: stopping tablet replica
I20260927 18:45:10.216140 4529 tablet_replica.cc:330] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: stopping tablet replica
I20260927 18:45:10.216606 4529 raft_consensus.cc:2229] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:10.217252 4529 raft_consensus.cc:2258] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:10.217290 4528 raft_consensus.cc:2258] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:10.216269 4527 raft_consensus.cc:2229] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:10.218047 4527 raft_consensus.cc:2258] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:10.218343 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.207002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.31.26.193:37367)
0927 18:45:10.207170 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:10.207176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:10.211172 (+ 3996us) server_negotiation.cc:407] Connection header received
0927 18:45:10.211401 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:10.211408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:10.211490 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:10.211604 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:10.217165 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:10.217172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:10.217177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:10.217204 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:10.217253 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:10.217282 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:10.217411 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:10.217959 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 18:45:10.218024 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:45:10.220013 4527 ts_tablet_manager.cc:1905] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:10.220013 4529 ts_tablet_manager.cc:1905] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:10.220192 4528 ts_tablet_manager.cc:1905] T bda93709eb444ab29af65b43c7659e77 P fdd8bacf1ee84dd1ac25ddfafb027eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:10.222025 4220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:10.227344 4527 ts_tablet_manager.cc:1918] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:10.227543 4527 log.cc:1196] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/bda93709eb444ab29af65b43c7659e77
I20260927 18:45:10.227973 4529 ts_tablet_manager.cc:1918] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:10.228075 4527 ts_tablet_manager.cc:1939] T bda93709eb444ab29af65b43c7659e77 P f07a8d574448488198e6b3976222fd90: Deleting consensus metadata
I20260927 18:45:10.228139 4529 log.cc:1196] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/bda93709eb444ab29af65b43c7659e77
I20260927 18:45:10.228632 4529 ts_tablet_manager.cc:1939] T bda93709eb444ab29af65b43c7659e77 P 9a24c41492b24e6d89e607500338fc81: Deleting consensus metadata
I20260927 18:45:10.230234 4026 catalog_manager.cc:4841] TS f07a8d574448488198e6b3976222fd90 (127.31.26.194:44351): tablet bda93709eb444ab29af65b43c7659e77 (table test-table [id=82835284f10541159e8f4a2a6f0fcbcb]) successfully deleted
I20260927 18:45:10.230315 4025 catalog_manager.cc:4841] TS 9a24c41492b24e6d89e607500338fc81 (127.31.26.195:34751): tablet bda93709eb444ab29af65b43c7659e77 (table test-table [id=82835284f10541159e8f4a2a6f0fcbcb]) successfully deleted
I20260927 18:45:10.316023 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4098
I20260927 18:45:10.333591 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4229
W20260927 18:45:10.333690 4026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.26.193:37367 (error 108)
I20260927 18:45:10.336010 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.335469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:42988)
0927 18:45:10.335680 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.335822 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:45:10.336265 4026 catalog_manager.cc:4584] TS fdd8bacf1ee84dd1ac25ddfafb027eb2 (127.31.26.193:37367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bda93709eb444ab29af65b43c7659e77: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
I20260927 18:45:10.336907 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.334545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.31.26.194:44351)
0927 18:45:10.334735 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:10.334742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:10.334832 (+ 90us) server_negotiation.cc:407] Connection header received
0927 18:45:10.334997 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:10.335003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:10.335114 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:10.335237 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:10.336318 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:10.336323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:10.336327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:10.336346 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:10.336384 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:10.336400 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:10.336532 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:10.336658 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:45:10.336704 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:45:10.337558 4351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:10.390000 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.389535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:51064)
0927 18:45:10.389743 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.389840 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 18:45:10.391069 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.390741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:51074)
0927 18:45:10.390870 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.390957 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 18:45:10.425434 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4229
I20260927 18:45:10.441982 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4359
I20260927 18:45:10.444869 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.442870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.31.26.195:34751)
0927 18:45:10.443054 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:10.443059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:10.443119 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:45:10.443300 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:10.443304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:10.443400 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:10.443488 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:10.444372 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:10.444376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:10.444379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:10.444396 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:10.444426 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:10.444438 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:10.444538 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:10.444593 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:45:10.444637 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:45:10.445559 4481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:10.452123 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.451681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:51076)
0927 18:45:10.451874 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.451974 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 18:45:10.453176 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.452761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:51086)
0927 18:45:10.452916 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.452993 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:45:10.533833 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.533367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:51102)
0927 18:45:10.533586 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.533685 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:45:10.534902 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:10.534490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37367 (local address 127.0.0.1:51106)
0927 18:45:10.534714 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:10.534787 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 18:45:10.548520 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4359
I20260927 18:45:10.565346 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4008
I20260927 18:45:10.566586 4069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:10.670951 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4008
2026-09-27T18:45:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2718 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 18:45:10.713598 31851 mini_oidc.cc:152] Starting JWKS server
I20260927 18:45:10.716092 31851 webserver.cc:460] Webserver started at https://127.0.0.1:44531/ using document root <none> and password file <none>
I20260927 18:45:10.716418 31851 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:45:10.717185 31851 webserver.cc:460] Webserver started at http://127.0.0.1:43869/ using document root <none> and password file <none>
2026-09-27T18:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:10Z Disabled control of system clock
I20260927 18:45:10.744412 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:37397
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:37397
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/testchainca.cert with env {}
W20260927 18:45:11.073513 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:11.073913 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:11.074083 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:11.081521 4541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:11.081703 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:11.081801 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:11.081919 4541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:11.082012 4541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:45:11.082103 4541 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:45:11.082183 4541 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44531/jwks.json
I20260927 18:45:11.092422 4541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:37397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:37397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:11.093847 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:11.095247 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:11.108078 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:11.108110 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:11.108117 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:11.108582 4541 server_base.cc:1034] running on GCE node
I20260927 18:45:11.109751 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:11.111370 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:11.112624 4541 hybrid_clock.cc:648] HybridClock initialized: now 1790534711112569 us; error 40 us; skew 500 ppm
I20260927 18:45:11.113173 4541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:11.115890 4541 webserver.cc:460] Webserver started at http://127.31.26.254:41575/ using document root <none> and password file <none>
I20260927 18:45:11.116734 4541 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:11.116870 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:11.117307 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:11.120617 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "04aaa78bfb4347b58325eb1589442565"
format_stamp: "Formatted at 2026-09-27 18:45:11 on dist-test-slave-rfd5"
server_key: "76bb778f91ec86edf4938a7d3437235f"
server_key_iv: "1323de7238084de0fb9f21aaff7c7d55"
server_key_version: "encryptionkey@0"
I20260927 18:45:11.121851 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "04aaa78bfb4347b58325eb1589442565"
format_stamp: "Formatted at 2026-09-27 18:45:11 on dist-test-slave-rfd5"
server_key: "76bb778f91ec86edf4938a7d3437235f"
server_key_iv: "1323de7238084de0fb9f21aaff7c7d55"
server_key_version: "encryptionkey@0"
I20260927 18:45:11.128496 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:45:11.132329 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:11.133939 4541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:11.134152 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "04aaa78bfb4347b58325eb1589442565"
format_stamp: "Formatted at 2026-09-27 18:45:11 on dist-test-slave-rfd5"
server_key: "76bb778f91ec86edf4938a7d3437235f"
server_key_iv: "1323de7238084de0fb9f21aaff7c7d55"
server_key_version: "encryptionkey@0"
I20260927 18:45:11.134423 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:11.168803 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:11.169795 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:11.170125 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:11.189421 4541 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:37397
I20260927 18:45:11.189445 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:37397 every 8 connection(s)
I20260927 18:45:11.190846 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:11.194036 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4541
I20260927 18:45:11.194267 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:11.194864 31851 external_mini_cluster.cc:1428] Setting key 5c915da5bbc6acc7deb9a0571e1d0975
I20260927 18:45:11.194928 4607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:11.210624 4607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565: Bootstrap starting.
I20260927 18:45:11.210930 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:11.196914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.31.26.254:37397)
0927 18:45:11.198033 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:11.198048 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:11.198105 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:45:11.200188 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:11.200228 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:11.200899 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:11.201441 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:11.202018 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.204048 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.205502 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.206557 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.206760 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:11.207910 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:11.207925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:11.207946 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:11.208716 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:11.208815 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:11.208842 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:11.208953 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:11.209219 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:45:11.209568 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260927 18:45:11.214234 4607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:11.214869 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:45:11.215483 4607 log.cc:824] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:11.218545 4607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565: No bootstrap required, opened a new log
I20260927 18:45:11.225737 4607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa78bfb4347b58325eb1589442565" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 37397 } }
I20260927 18:45:11.226117 4607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:11.226217 4607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04aaa78bfb4347b58325eb1589442565, State: Initialized, Role: FOLLOWER
I20260927 18:45:11.226876 4607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa78bfb4347b58325eb1589442565" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 37397 } }
I20260927 18:45:11.227108 4607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:11.227268 4607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:11.227442 4607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:11.229710 4607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa78bfb4347b58325eb1589442565" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 37397 } }
I20260927 18:45:11.230270 4607 leader_election.cc:302] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04aaa78bfb4347b58325eb1589442565; no voters:
I20260927 18:45:11.230827 4607 leader_election.cc:288] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:11.230998 4612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:11.231701 4612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [term 1 LEADER]: Becoming Leader. State: Replica: 04aaa78bfb4347b58325eb1589442565, State: Running, Role: LEADER
I20260927 18:45:11.232298 4612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa78bfb4347b58325eb1589442565" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 37397 } }
I20260927 18:45:11.232972 4607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:11.235647 4613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04aaa78bfb4347b58325eb1589442565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa78bfb4347b58325eb1589442565" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 37397 } } }
I20260927 18:45:11.235759 4614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04aaa78bfb4347b58325eb1589442565. Latest consensus state: current_term: 1 leader_uuid: "04aaa78bfb4347b58325eb1589442565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04aaa78bfb4347b58325eb1589442565" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 37397 } } }
I20260927 18:45:11.236042 4613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:11.236163 4614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:11.237180 4619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:11.244899 4619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:11.254774 4619 catalog_manager.cc:1347] Generated new cluster ID: 1a858be2cb4f4fddafcaa9e8241f5713
I20260927 18:45:11.254906 4619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:11.277827 4619 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:11.279237 4619 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:11.288453 4619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04aaa78bfb4347b58325eb1589442565: Generated new TSK 0
I20260927 18:45:11.289515 4619 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:45:11.331619 4637 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 18:45:11.331831 4637 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:45:11.333022 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:11.310517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.31.26.254:37397)
0927 18:45:11.310732 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:11.310740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:11.311294 (+ 554us) server_negotiation.cc:407] Connection header received
0927 18:45:11.311612 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:11.311618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:11.311712 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:11.311890 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:45:11.312423 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.313634 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.315225 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.315932 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.316071 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:11.316669 (+ 598us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:45:11.332297 (+ 15628us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:45:11.332679 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 18:45:11.333168 4610 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 18:45:11.334755 4636 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:11.310418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:37397 (local address 127.0.0.1:49862)
0927 18:45:11.311179 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:11.311196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:11.311368 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:11.311924 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:11.311928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:11.311939 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:45:11.312264 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:11.312272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.313864 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.313869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:11.315071 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:11.315083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.316133 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.316138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:11.316261 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:11.316344 (+ 83us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:45:11.332599 (+ 16255us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:45:11.333002 (+ 403us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.26.254:37397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":413,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.26.254:37397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 18:45:11.337020 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4541
I20260927 18:45:11.338291 4602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:11.432739 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4541
2026-09-27T18:45:11Z chronyd exiting
I20260927 18:45:11.484148 31851 test_util.cc:175] -----------------------------------------------
I20260927 18:45:11.484292 31851 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534668899567-31851-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (775 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 18:45:11.488101 31851 mini_oidc.cc:152] Starting JWKS server
I20260927 18:45:11.490097 31851 webserver.cc:460] Webserver started at https://127.0.0.1:40309/ using document root <none> and password file <none>
I20260927 18:45:11.490288 31851 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:45:11.490933 31851 webserver.cc:460] Webserver started at http://127.0.0.1:45563/ using document root <none> and password file <none>
2026-09-27T18:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:11Z Disabled control of system clock
I20260927 18:45:11.518344 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:40351
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:40351
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40309/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/testchainca.cert with env {}
W20260927 18:45:11.848407 4644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:11.848814 4644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:11.848987 4644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:11.856415 4644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:11.856594 4644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:11.856699 4644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:11.856801 4644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:11.856884 4644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:45:11.856977 4644 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:45:11.857059 4644 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40309/jwks.json
I20260927 18:45:11.867343 4644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:40351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40309/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:11.868769 4644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:11.870152 4644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:11.882517 4649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:11.882455 4650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:11.882578 4652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:11.883585 4644 server_base.cc:1034] running on GCE node
I20260927 18:45:11.884493 4644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:11.886092 4644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:11.887364 4644 hybrid_clock.cc:648] HybridClock initialized: now 1790534711887299 us; error 57 us; skew 500 ppm
I20260927 18:45:11.887780 4644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:11.890440 4644 webserver.cc:460] Webserver started at http://127.31.26.254:42563/ using document root <none> and password file <none>
I20260927 18:45:11.891211 4644 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:11.891350 4644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:11.891681 4644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:11.894740 4644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "173bfd7bb1bb4895820e0db58a8bbe76"
format_stamp: "Formatted at 2026-09-27 18:45:11 on dist-test-slave-rfd5"
server_key: "1051022b25cfe1afcbff3d67312a5d51"
server_key_iv: "d3537c48b95aaafbb5f01ebf7e7960e7"
server_key_version: "encryptionkey@0"
I20260927 18:45:11.895622 4644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "173bfd7bb1bb4895820e0db58a8bbe76"
format_stamp: "Formatted at 2026-09-27 18:45:11 on dist-test-slave-rfd5"
server_key: "1051022b25cfe1afcbff3d67312a5d51"
server_key_iv: "d3537c48b95aaafbb5f01ebf7e7960e7"
server_key_version: "encryptionkey@0"
I20260927 18:45:11.901181 4644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:45:11.904772 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:11.906338 4644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:45:11.906545 4644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "173bfd7bb1bb4895820e0db58a8bbe76"
format_stamp: "Formatted at 2026-09-27 18:45:11 on dist-test-slave-rfd5"
server_key: "1051022b25cfe1afcbff3d67312a5d51"
server_key_iv: "d3537c48b95aaafbb5f01ebf7e7960e7"
server_key_version: "encryptionkey@0"
I20260927 18:45:11.906790 4644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:11.931728 4644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:11.932668 4644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:11.933009 4644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:11.951865 4644 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:40351
I20260927 18:45:11.951886 4709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:40351 every 8 connection(s)
I20260927 18:45:11.953354 4644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:11.957577 4710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:11.958024 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4644
I20260927 18:45:11.958223 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:11.959045 31851 external_mini_cluster.cc:1428] Setting key 3a7b28010fe5cb85e1d5174d1b00777b
I20260927 18:45:11.971127 4710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76: Bootstrap starting.
I20260927 18:45:11.973922 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:11.961212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.31.26.254:40351)
0927 18:45:11.962414 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:11.962431 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:11.962486 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:45:11.964458 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:11.964487 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:11.965151 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:11.965664 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:11.966290 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.967626 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.969001 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:11.969839 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:11.970086 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:11.971405 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:11.971414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:11.971431 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:11.972015 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:11.972091 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:11.972113 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:11.972246 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:11.972513 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 18:45:11.972839 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":404,"threads_started":1}
I20260927 18:45:11.974798 4710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:11.975945 4710 log.cc:824] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:11.978297 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:45:11.979446 4710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76: No bootstrap required, opened a new log
I20260927 18:45:11.987279 4710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 40351 } }
I20260927 18:45:11.987653 4710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:11.987728 4710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173bfd7bb1bb4895820e0db58a8bbe76, State: Initialized, Role: FOLLOWER
I20260927 18:45:11.988372 4710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 40351 } }
I20260927 18:45:11.988629 4710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:11.988786 4710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:11.988965 4710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:11.991149 4710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 40351 } }
I20260927 18:45:11.991652 4710 leader_election.cc:302] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 173bfd7bb1bb4895820e0db58a8bbe76; no voters:
I20260927 18:45:11.992231 4710 leader_election.cc:288] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:11.992388 4715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:11.993127 4715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [term 1 LEADER]: Becoming Leader. State: Replica: 173bfd7bb1bb4895820e0db58a8bbe76, State: Running, Role: LEADER
I20260927 18:45:11.993700 4715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 40351 } }
I20260927 18:45:11.994428 4710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:11.996384 4716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 40351 } } }
I20260927 18:45:11.996841 4716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:11.996762 4717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173bfd7bb1bb4895820e0db58a8bbe76. Latest consensus state: current_term: 1 leader_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173bfd7bb1bb4895820e0db58a8bbe76" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 40351 } } }
I20260927 18:45:11.997149 4717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:11.998281 4720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:12.005806 4720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:12.014987 4720 catalog_manager.cc:1347] Generated new cluster ID: 1d3b35492236433182525e463f12165c
I20260927 18:45:12.015116 4720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:12.046304 4720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:12.047927 4720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:12.056802 4720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173bfd7bb1bb4895820e0db58a8bbe76: Generated new TSK 0
I20260927 18:45:12.057773 4720 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:45:12.089500 4740 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 18:45:12.089644 4740 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:45:12.091140 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:12.071102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.31.26.254:40351)
0927 18:45:12.071263 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:12.071268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:12.071761 (+ 493us) server_negotiation.cc:407] Connection header received
0927 18:45:12.072026 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:12.072031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:12.072105 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:12.072271 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:45:12.072748 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.073553 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.074931 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.075391 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.075522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:12.076168 (+ 646us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:45:12.090390 (+ 14222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:45:12.090818 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 18:45:12.091238 4739 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:12.071003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:40351 (local address 127.0.0.1:46160)
0927 18:45:12.071649 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:12.071664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:12.071833 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:12.072281 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:12.072286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:12.072297 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:45:12.072576 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:12.072586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.073700 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.073706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:12.074780 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:12.074791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.075563 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.075571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:12.075788 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:12.075844 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:45:12.090664 (+ 14820us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:45:12.090869 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.26.254:40351: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":401,"threads_started":1}
W20260927 18:45:12.091269 4713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40309/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 18:45:12.091944 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4644
I20260927 18:45:12.093235 4705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:12.196851 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4644
2026-09-27T18:45:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (764 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 18:45:12.253137 31851 mini_oidc.cc:152] Starting JWKS server
I20260927 18:45:12.255343 31851 webserver.cc:460] Webserver started at https://127.0.0.1:45655/ using document root <none> and password file <none>
I20260927 18:45:12.255620 31851 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:45:12.256331 31851 webserver.cc:460] Webserver started at http://127.0.0.1:35379/ using document root <none> and password file <none>
2026-09-27T18:45:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:12Z Disabled control of system clock
I20260927 18:45:12.284771 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33629
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:33629
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:45:12.615763 4747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:12.616171 4747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:12.616341 4747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:12.623796 4747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:12.623980 4747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:12.624076 4747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:12.624184 4747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:12.624258 4747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:45:12.624356 4747 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:45:12.624431 4747 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45655/jwks.json
I20260927 18:45:12.634119 4747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:33629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:12.635533 4747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:12.637617 4747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:12.651767 4753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:12.651858 4755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:12.651793 4752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:12.653411 4747 server_base.cc:1034] running on GCE node
I20260927 18:45:12.654354 4747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:12.656035 4747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:12.657294 4747 hybrid_clock.cc:648] HybridClock initialized: now 1790534712657243 us; error 36 us; skew 500 ppm
I20260927 18:45:12.657702 4747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:12.660288 4747 webserver.cc:460] Webserver started at http://127.31.26.254:40183/ using document root <none> and password file <none>
I20260927 18:45:12.661063 4747 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:12.661245 4747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:12.661604 4747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:12.664577 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "ed5ef7908815445e9ebda1b0db1fe4db"
format_stamp: "Formatted at 2026-09-27 18:45:12 on dist-test-slave-rfd5"
server_key: "960e06f03b9dba8fa9b7dea0d92e47e5"
server_key_iv: "355e7e72efefaf8584f7ccd22285a1cf"
server_key_version: "encryptionkey@0"
I20260927 18:45:12.665454 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "ed5ef7908815445e9ebda1b0db1fe4db"
format_stamp: "Formatted at 2026-09-27 18:45:12 on dist-test-slave-rfd5"
server_key: "960e06f03b9dba8fa9b7dea0d92e47e5"
server_key_iv: "355e7e72efefaf8584f7ccd22285a1cf"
server_key_version: "encryptionkey@0"
I20260927 18:45:12.670817 4747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 18:45:12.674496 4761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:12.675949 4747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:12.676165 4747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "ed5ef7908815445e9ebda1b0db1fe4db"
format_stamp: "Formatted at 2026-09-27 18:45:12 on dist-test-slave-rfd5"
server_key: "960e06f03b9dba8fa9b7dea0d92e47e5"
server_key_iv: "355e7e72efefaf8584f7ccd22285a1cf"
server_key_version: "encryptionkey@0"
I20260927 18:45:12.676397 4747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:12.700327 4747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:12.701251 4747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:12.701565 4747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:12.720227 4747 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:33629
I20260927 18:45:12.720235 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:33629 every 8 connection(s)
I20260927 18:45:12.721657 4747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:12.725590 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:12.725682 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4747
I20260927 18:45:12.725903 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:12.726537 31851 external_mini_cluster.cc:1428] Setting key bc242cda11b790a5839df48af3046dcf
I20260927 18:45:12.737987 4813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db: Bootstrap starting.
I20260927 18:45:12.740531 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:12.728456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.31.26.254:33629)
0927 18:45:12.729604 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:12.729618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:12.729664 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:45:12.731750 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:12.731776 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:12.732299 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:12.732753 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:12.733342 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.734619 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.735993 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.736687 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.736885 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:12.738166 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:12.738177 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:12.738197 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:12.738754 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:12.738831 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:12.738854 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:12.738985 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:12.739225 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 18:45:12.739564 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":383,"threads_started":1}
I20260927 18:45:12.741528 4813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:12.742676 4813 log.cc:824] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:12.744730 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:45:12.745810 4813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db: No bootstrap required, opened a new log
I20260927 18:45:12.753811 4813 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33629 } }
I20260927 18:45:12.754207 4813 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:12.754371 4813 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5ef7908815445e9ebda1b0db1fe4db, State: Initialized, Role: FOLLOWER
I20260927 18:45:12.754973 4813 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33629 } }
I20260927 18:45:12.755194 4813 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:12.755354 4813 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:12.755537 4813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:12.758090 4813 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33629 } }
I20260927 18:45:12.758643 4813 leader_election.cc:302] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5ef7908815445e9ebda1b0db1fe4db; no voters:
I20260927 18:45:12.759260 4813 leader_election.cc:288] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:12.759421 4818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:12.760200 4818 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [term 1 LEADER]: Becoming Leader. State: Replica: ed5ef7908815445e9ebda1b0db1fe4db, State: Running, Role: LEADER
I20260927 18:45:12.760753 4818 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33629 } }
I20260927 18:45:12.761569 4813 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:12.763783 4819 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33629 } } }
I20260927 18:45:12.764266 4819 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:12.764803 4820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5ef7908815445e9ebda1b0db1fe4db. Latest consensus state: current_term: 1 leader_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ef7908815445e9ebda1b0db1fe4db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33629 } } }
I20260927 18:45:12.765159 4820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:12.765913 4827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:12.770958 4827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:12.780360 4827 catalog_manager.cc:1347] Generated new cluster ID: 41fe94dae2a2472086fb5a012db3c3ce
I20260927 18:45:12.780498 4827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:12.827667 4827 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:12.829020 4827 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:12.838805 4827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5ef7908815445e9ebda1b0db1fe4db: Generated new TSK 0
I20260927 18:45:12.839730 4827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:45:12.868808 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:12.859766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.31.26.254:33629)
0927 18:45:12.859965 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:12.859971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:12.860304 (+ 333us) server_negotiation.cc:407] Connection header received
0927 18:45:12.860523 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:12.860530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:12.860621 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:12.860789 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:45:12.861377 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.862531 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.864071 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.864772 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.864919 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:12.865667 (+ 748us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:45:12.868139 (+ 2472us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:45:12.868477 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:45:12.868954 4816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 18:45:12.870618 4842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:12.859566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33629 (local address 127.0.0.1:60996)
0927 18:45:12.860195 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:12.860210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:12.860330 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:12.860828 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:12.860832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:12.860842 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:45:12.861230 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:12.861241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.862740 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.862744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:12.863913 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:12.863925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:12.865078 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:12.865086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:12.865294 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:12.865354 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:45:12.868367 (+ 3013us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:45:12.870183 (+ 1816us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.26.254:33629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":1490,"thread_start_us":327,"threads_started":1}
I20260927 18:45:12.871694 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4747
I20260927 18:45:12.872957 4808 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 18:45:12.881021 4843 webserver.cc:533] Webserver: sslize failed:
E20260927 18:45:12.881193 4843 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:45:12.967025 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4747
2026-09-27T18:45:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (770 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T18:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:13Z Disabled control of system clock
I20260927 18:45:13.050508 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35109
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35109
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/testchainca.cert with env {}
W20260927 18:45:13.387027 4848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:13.387368 4848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:13.387463 4848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:13.395154 4848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:13.395292 4848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:13.395331 4848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:13.395382 4848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:13.395426 4848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:45:13.395483 4848 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:45:13.395521 4848 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 18:45:13.406010 4848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:13.407323 4848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:13.408607 4848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:13.421166 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:13.421244 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:13.421298 4856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:13.422035 4848 server_base.cc:1034] running on GCE node
I20260927 18:45:13.422904 4848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:13.424434 4848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:13.425669 4848 hybrid_clock.cc:648] HybridClock initialized: now 1790534713425625 us; error 35 us; skew 500 ppm
I20260927 18:45:13.426046 4848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:45:13.428572 4848 webserver.cc:460] Webserver started at http://127.31.26.254:38523/ using document root <none> and password file <none>
I20260927 18:45:13.429427 4848 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:13.429570 4848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:13.429883 4848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:13.433027 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "af53d22461274e82948e173fa19a2398"
format_stamp: "Formatted at 2026-09-27 18:45:13 on dist-test-slave-rfd5"
server_key: "4de252ba6c1482b0764d5446acb04e31"
server_key_iv: "c34f398b945fc3bfa7ae8c501ccbb524"
server_key_version: "encryptionkey@0"
I20260927 18:45:13.434028 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "af53d22461274e82948e173fa19a2398"
format_stamp: "Formatted at 2026-09-27 18:45:13 on dist-test-slave-rfd5"
server_key: "4de252ba6c1482b0764d5446acb04e31"
server_key_iv: "c34f398b945fc3bfa7ae8c501ccbb524"
server_key_version: "encryptionkey@0"
I20260927 18:45:13.439620 4848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:45:13.443181 4862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:13.444545 4848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:13.444742 4848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "af53d22461274e82948e173fa19a2398"
format_stamp: "Formatted at 2026-09-27 18:45:13 on dist-test-slave-rfd5"
server_key: "4de252ba6c1482b0764d5446acb04e31"
server_key_iv: "c34f398b945fc3bfa7ae8c501ccbb524"
server_key_version: "encryptionkey@0"
I20260927 18:45:13.444969 4848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:13.460659 4848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:13.461777 4848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:13.462117 4848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:13.481472 4848 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:35109
I20260927 18:45:13.481490 4913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:35109 every 8 connection(s)
I20260927 18:45:13.482985 4848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:13.487247 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:13.491976 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4848
I20260927 18:45:13.492195 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:13.492841 31851 external_mini_cluster.cc:1428] Setting key 67c87890463ea89a5c677e6c869a641b
I20260927 18:45:13.499310 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398: Bootstrap starting.
I20260927 18:45:13.502744 4914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:13.503840 4914 log.cc:824] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:13.507098 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398: No bootstrap required, opened a new log
I20260927 18:45:13.507387 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:13.494918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.31.26.254:35109)
0927 18:45:13.495947 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:13.495967 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:13.496012 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:45:13.497895 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:13.497921 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:13.498418 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:13.498834 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:13.499439 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:13.500804 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:13.502354 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:13.503188 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:13.503386 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:13.504762 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:13.504770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:13.504785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:45:13.505551 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:13.505632 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:45:13.505656 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:13.505756 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:13.506011 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:45:13.506389 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":333,"threads_started":1}
I20260927 18:45:13.511634 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:45:13.516171 4914 raft_consensus.cc:348] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af53d22461274e82948e173fa19a2398" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35109 } }
I20260927 18:45:13.516561 4914 raft_consensus.cc:374] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:13.516685 4914 raft_consensus.cc:729] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af53d22461274e82948e173fa19a2398, State: Initialized, Role: FOLLOWER
I20260927 18:45:13.517302 4914 consensus_queue.cc:260] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af53d22461274e82948e173fa19a2398" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35109 } }
I20260927 18:45:13.517540 4914 raft_consensus.cc:388] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:13.517699 4914 raft_consensus.cc:482] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:13.517870 4914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:13.520087 4914 raft_consensus.cc:504] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af53d22461274e82948e173fa19a2398" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35109 } }
I20260927 18:45:13.520690 4914 leader_election.cc:302] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af53d22461274e82948e173fa19a2398; no voters:
I20260927 18:45:13.521514 4914 leader_election.cc:288] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:13.521660 4919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:13.522495 4919 raft_consensus.cc:686] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [term 1 LEADER]: Becoming Leader. State: Replica: af53d22461274e82948e173fa19a2398, State: Running, Role: LEADER
I20260927 18:45:13.523142 4919 consensus_queue.cc:237] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af53d22461274e82948e173fa19a2398" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35109 } }
I20260927 18:45:13.523774 4914 sys_catalog.cc:564] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:13.526561 4920 sys_catalog.cc:455] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af53d22461274e82948e173fa19a2398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af53d22461274e82948e173fa19a2398" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35109 } } }
I20260927 18:45:13.526582 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af53d22461274e82948e173fa19a2398. Latest consensus state: current_term: 1 leader_uuid: "af53d22461274e82948e173fa19a2398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af53d22461274e82948e173fa19a2398" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35109 } } }
I20260927 18:45:13.526988 4920 sys_catalog.cc:458] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:13.526988 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:13.527942 4927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:13.534979 4927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:13.545306 4927 catalog_manager.cc:1347] Generated new cluster ID: ac35af9670c54b2eb8889a2b2d8eb56e
I20260927 18:45:13.545446 4927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:13.570703 4927 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:13.572578 4927 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:13.581240 4927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af53d22461274e82948e173fa19a2398: Generated new TSK 0
I20260927 18:45:13.582296 4927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:45:13.669219 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:13.604491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.31.26.254:35109)
0927 18:45:13.604743 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:13.604750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:13.605385 (+ 635us) server_negotiation.cc:407] Connection header received
0927 18:45:13.605666 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:13.605674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:13.605785 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:13.605968 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:45:13.606417 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:13.607595 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:13.609005 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:13.609755 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:13.609887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:13.610395 (+ 508us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:45:13.668255 (+ 57860us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:45:13.668792 (+ 537us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 18:45:13.669387 4943 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:13.604442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35109 (local address 127.0.0.1:53446)
0927 18:45:13.605265 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:13.605283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:13.605462 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:13.605980 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:13.605983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:13.605993 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:45:13.606269 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:13.606277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:13.607757 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:13.607761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:13.608857 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:13.608878 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:13.609939 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:13.609944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:13.610071 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:13.610105 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:45:13.668622 (+ 58517us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:45:13.668911 (+ 289us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.26.254:35109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":474,"threads_started":1}
W20260927 18:45:13.669410 4917 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 18:45:13.670727 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4848
I20260927 18:45:13.671973 4909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:13.771878 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4848
2026-09-27T18:45:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (783 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (423 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (424 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:14 dist-test-slave-rfd5 krb5kdc[4958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:14 dist-test-slave-rfd5 krb5kdc[4958](info): set up 2 sockets
Sep 27 18:45:14 dist-test-slave-rfd5 krb5kdc[4958](info): commencing operation
krb5kdc: starting...
W20260927 18:45:15.714363 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:15 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:45:15.924742 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34367
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:34367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:16.249810 4971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:16.250181 4971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:16.250345 4971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:16.257797 4971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:16.257962 4971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:16.258075 4971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:16.258178 4971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:16.258255 4971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:16.268277 4971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:34367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:16.269764 4971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:16.271097 4971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:16.283299 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:16.283505 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:16.283258 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:16.284375 4971 server_base.cc:1034] running on GCE node
I20260927 18:45:16.285367 4971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:16.286839 4971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:16.288067 4971 hybrid_clock.cc:648] HybridClock initialized: now 1790534716288011 us; error 44 us; skew 500 ppm
I20260927 18:45:16.288450 4971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:16 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:16.306428 4971 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:45:16.308123 4971 webserver.cc:460] Webserver started at http://127.31.26.254:46461/ using document root <none> and password file <none>
I20260927 18:45:16.308905 4971 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:16.309048 4971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:16.309531 4971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:16.312565 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "fd1113a722344eba99bd90f2600de644"
format_stamp: "Formatted at 2026-09-27 18:45:16 on dist-test-slave-rfd5"
server_key: "5f36af58b6188ac16949015119a02a8d"
server_key_iv: "59a88f19b6d78e2989c3f79969e706f4"
server_key_version: "encryptionkey@0"
I20260927 18:45:16.313469 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "fd1113a722344eba99bd90f2600de644"
format_stamp: "Formatted at 2026-09-27 18:45:16 on dist-test-slave-rfd5"
server_key: "5f36af58b6188ac16949015119a02a8d"
server_key_iv: "59a88f19b6d78e2989c3f79969e706f4"
server_key_version: "encryptionkey@0"
I20260927 18:45:16.318998 4971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:45:16.322641 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:16.324077 4971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:45:16.324285 4971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "fd1113a722344eba99bd90f2600de644"
format_stamp: "Formatted at 2026-09-27 18:45:16 on dist-test-slave-rfd5"
server_key: "5f36af58b6188ac16949015119a02a8d"
server_key_iv: "59a88f19b6d78e2989c3f79969e706f4"
server_key_version: "encryptionkey@0"
I20260927 18:45:16.324498 4971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:16.343369 4971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:16.348189 4971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:16.348501 4971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:16.366937 4971 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:34367
I20260927 18:45:16.366961 5037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:34367 every 8 connection(s)
I20260927 18:45:16.368348 4971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:16.372048 5038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:16.375123 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 4971
I20260927 18:45:16.375319 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:16.375982 31851 external_mini_cluster.cc:1428] Setting key 751c85729c32a0eb43632b7b338a00a7
I20260927 18:45:16.385142 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644: Bootstrap starting.
I20260927 18:45:16.388821 5038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:16.389870 5038 log.cc:824] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:45:16 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:16.393009 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644: No bootstrap required, opened a new log
I20260927 18:45:16.400604 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:16.378079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.31.26.254:34367)
0927 18:45:16.378891 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:16.378907 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:16.378958 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:45:16.380924 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:16.380953 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:16.381516 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:16.381971 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:16.383243 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:16.385196 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:16.386472 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:16.387561 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:16.387748 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:16.391938 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:16.391952 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:16.391970 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:16.392023 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:16.395990 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:16.396726 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:16.396735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:16.396743 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:16.396945 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:16.397362 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:16.397367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:16.397370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:16.398079 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:16.398383 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:16.398772 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:45:16.399221 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260927 18:45:16.403653 5038 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1113a722344eba99bd90f2600de644" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34367 } }
I20260927 18:45:16.403990 5038 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:16.404062 5038 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1113a722344eba99bd90f2600de644, State: Initialized, Role: FOLLOWER
I20260927 18:45:16.404521 5038 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1113a722344eba99bd90f2600de644" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34367 } }
I20260927 18:45:16.404716 5038 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:16.404840 5038 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:16.405027 5038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:16.407316 5038 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1113a722344eba99bd90f2600de644" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34367 } }
I20260927 18:45:16.407747 5038 leader_election.cc:302] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1113a722344eba99bd90f2600de644; no voters:
I20260927 18:45:16.408352 5038 leader_election.cc:288] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:16.408510 5043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:16.409232 5043 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [term 1 LEADER]: Becoming Leader. State: Replica: fd1113a722344eba99bd90f2600de644, State: Running, Role: LEADER
I20260927 18:45:16.409886 5043 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1113a722344eba99bd90f2600de644" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34367 } }
I20260927 18:45:16.410408 5038 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:16.413160 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1113a722344eba99bd90f2600de644. Latest consensus state: current_term: 1 leader_uuid: "fd1113a722344eba99bd90f2600de644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1113a722344eba99bd90f2600de644" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34367 } } }
I20260927 18:45:16.413579 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:16.413233 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1113a722344eba99bd90f2600de644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1113a722344eba99bd90f2600de644" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34367 } } }
I20260927 18:45:16.413866 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:16.417928 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:16.424505 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:16.436542 5049 catalog_manager.cc:1347] Generated new cluster ID: c7bb57caefdb43ea8a1e3d0e50318032
I20260927 18:45:16.436753 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:16.456164 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:16.457576 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:16.464950 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1113a722344eba99bd90f2600de644: Generated new TSK 0
I20260927 18:45:16.466209 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:45:16.528870 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34367
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:16.898808 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:16.899151 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:16.899315 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:16.906953 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:16.907137 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:16.907320 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:16.917886 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:16.919286 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:16.920674 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:16.935215 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:16.935333 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:16.935261 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:16.936409 5064 server_base.cc:1034] running on GCE node
I20260927 18:45:16.937252 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:16.938402 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:16.939692 5064 hybrid_clock.cc:648] HybridClock initialized: now 1790534716939622 us; error 54 us; skew 500 ppm
I20260927 18:45:16.940049 5064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:16 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:16.943550 5064 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:45:16.945302 5064 webserver.cc:460] Webserver started at http://127.31.26.193:40575/ using document root <none> and password file <none>
I20260927 18:45:16.946048 5064 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:16.946206 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:16.946527 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:16.949600 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "b6968de4f25f428cbdd0623516a6f433"
format_stamp: "Formatted at 2026-09-27 18:45:16 on dist-test-slave-rfd5"
server_key: "e89e867dc6092bc1c1eb9e4e5c166862"
server_key_iv: "9d40e0f45ce89d097081a8b3f1706780"
server_key_version: "encryptionkey@0"
I20260927 18:45:16.950416 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "b6968de4f25f428cbdd0623516a6f433"
format_stamp: "Formatted at 2026-09-27 18:45:16 on dist-test-slave-rfd5"
server_key: "e89e867dc6092bc1c1eb9e4e5c166862"
server_key_iv: "9d40e0f45ce89d097081a8b3f1706780"
server_key_version: "encryptionkey@0"
I20260927 18:45:16.955927 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:45:16.959702 5079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:16.961283 5064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:45:16.961475 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "b6968de4f25f428cbdd0623516a6f433"
format_stamp: "Formatted at 2026-09-27 18:45:16 on dist-test-slave-rfd5"
server_key: "e89e867dc6092bc1c1eb9e4e5c166862"
server_key_iv: "9d40e0f45ce89d097081a8b3f1706780"
server_key_version: "encryptionkey@0"
I20260927 18:45:16.961701 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:16.979506 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:16.984062 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:16.984437 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:16.985567 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:16.987044 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:16.987164 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:16.987340 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:16.987424 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:17.022840 5064 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:40021
I20260927 18:45:17.022893 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:40021 every 8 connection(s)
I20260927 18:45:17.024281 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:17.033895 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5064
I20260927 18:45:17.034102 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:17.034924 31851 external_mini_cluster.cc:1428] Setting key c2b4ac57ec2301ebebc1b464763c4248
Sep 27 18:45:17 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534716, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:17.047053 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:17.027045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:60165 (local address 127.31.26.254:34367)
0927 18:45:17.027548 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:17.027553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:17.028760 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 18:45:17.030702 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:17.030706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:17.030776 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:17.030890 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:17.032491 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.033347 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.035839 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.036420 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.036558 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:17.041002 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:17.041008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:17.041011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:17.041046 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:17.043782 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:17.044616 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:17.044624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:17.044628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:17.044771 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:17.045283 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:17.045289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:17.045294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:17.045579 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:17.045740 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:17.046470 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 18:45:17.046697 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
I20260927 18:45:17.048206 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:17.027739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34367 (local address 127.31.26.193:60165)
0927 18:45:17.028578 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:17.028630 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:17.030413 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:17.031144 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:17.031155 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:17.031724 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:17.032308 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:17.032327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.033517 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.033524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:17.035660 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:17.035674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.036653 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.036659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:17.036986 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:17.037785 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:17.037835 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:17.040720 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:17.043975 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:17.043985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:17.044012 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:17.044470 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:17.044895 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:17.044901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:17.044905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:17.045168 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:17.045715 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:17.045725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:17.046098 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 18:45:17.046377 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 18:45:17.046937 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":332,"threads_started":1}
I20260927 18:45:17.049960 5192 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34367
I20260927 18:45:17.050318 5192 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:17.051371 5192 heartbeater.cc:511] Master 127.31.26.254:34367 requested a full tablet report, sending...
I20260927 18:45:17.054201 5003 ts_manager.cc:194] Registered new tserver with Master: b6968de4f25f428cbdd0623516a6f433 (127.31.26.193:40021)
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:45:17.056205 5003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:60165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:45:17.084771 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34367
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:17.420400 5198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:17.420755 5198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:17.420928 5198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:17.428521 5198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:17.428685 5198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:17.428870 5198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:17.439613 5198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:17.440881 5198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:17.442166 5198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:17.455997 5203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:17.456027 5206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:17.455981 5204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:17.456923 5198 server_base.cc:1034] running on GCE node
I20260927 18:45:17.457623 5198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:17.458585 5198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:17.459815 5198 hybrid_clock.cc:648] HybridClock initialized: now 1790534717459771 us; error 45 us; skew 500 ppm
I20260927 18:45:17.460142 5198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:17 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534717, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:17.463321 5198 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:45:17.464819 5198 webserver.cc:460] Webserver started at http://127.31.26.194:42139/ using document root <none> and password file <none>
I20260927 18:45:17.465678 5198 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:17.465814 5198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:17.466226 5198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:17.469388 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "5a56d09d980d4cda86216b64a510bee1"
format_stamp: "Formatted at 2026-09-27 18:45:17 on dist-test-slave-rfd5"
server_key: "20b0b7dbfaf4a2848e8919810aeac13e"
server_key_iv: "89a3c283e439aa6686c07697f380f1f3"
server_key_version: "encryptionkey@0"
I20260927 18:45:17.470273 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "5a56d09d980d4cda86216b64a510bee1"
format_stamp: "Formatted at 2026-09-27 18:45:17 on dist-test-slave-rfd5"
server_key: "20b0b7dbfaf4a2848e8919810aeac13e"
server_key_iv: "89a3c283e439aa6686c07697f380f1f3"
server_key_version: "encryptionkey@0"
I20260927 18:45:17.477530 5198 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 18:45:17.481348 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:17.482722 5198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:17.482919 5198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "5a56d09d980d4cda86216b64a510bee1"
format_stamp: "Formatted at 2026-09-27 18:45:17 on dist-test-slave-rfd5"
server_key: "20b0b7dbfaf4a2848e8919810aeac13e"
server_key_iv: "89a3c283e439aa6686c07697f380f1f3"
server_key_version: "encryptionkey@0"
I20260927 18:45:17.483137 5198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:17.503259 5198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:17.507848 5198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:17.508159 5198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:17.509284 5198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:17.510991 5198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:17.511096 5198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:17.511255 5198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:17.511328 5198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:17.546072 5198 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:37181
I20260927 18:45:17.546170 5325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:37181 every 8 connection(s)
I20260927 18:45:17.547410 5198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:17.556896 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5198
I20260927 18:45:17.557117 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:17.557677 31851 external_mini_cluster.cc:1428] Setting key 0a9a9df1d0de88aea4a333ab20c0eb14
Sep 27 18:45:17 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534717, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:17.569397 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:17.550120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:33441 (local address 127.31.26.254:34367)
0927 18:45:17.550566 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:17.550570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:17.551661 (+ 1091us) server_negotiation.cc:407] Connection header received
0927 18:45:17.553585 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:17.553591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:17.553687 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:17.553817 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:17.555399 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.556141 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.558509 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.559023 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.559162 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:17.563299 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:17.563304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:17.563307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:17.563345 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:17.566080 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:17.566950 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:17.566956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:17.566960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:17.567096 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:17.567557 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:17.567566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:17.567568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:17.567850 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:17.568007 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:17.568752 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 18:45:17.568993 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260927 18:45:17.570523 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:17.550729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34367 (local address 127.31.26.194:33441)
0927 18:45:17.551480 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:17.551527 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:17.553294 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:17.554084 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:17.554093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:17.554630 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:17.555208 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:17.555226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.556322 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.556326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:17.558355 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:17.558369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:17.559262 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:17.559270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:17.559519 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:17.560246 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:17.560277 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:17.563048 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:17.566268 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:17.566278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:17.566313 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:17.566806 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:17.567212 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:17.567218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:17.567222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:17.567430 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:17.567983 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:17.567991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:17.568374 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 18:45:17.568660 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 18:45:17.569268 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":285,"threads_started":1}
I20260927 18:45:17.572247 5326 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34367
I20260927 18:45:17.572574 5326 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:17.573403 5326 heartbeater.cc:511] Master 127.31.26.254:34367 requested a full tablet report, sending...
I20260927 18:45:17.574983 5003 ts_manager.cc:194] Registered new tserver with Master: 5a56d09d980d4cda86216b64a510bee1 (127.31.26.194:37181)
I20260927 18:45:17.575834 5003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:33441
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:45:17.605607 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34367
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:17.938833 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:17.939127 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:17.939229 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:17.946318 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:17.946439 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:17.946630 5332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:17.956252 5332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:17.957537 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:17.958798 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:17.972321 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:17.972257 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:17.972326 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:17.973179 5332 server_base.cc:1034] running on GCE node
I20260927 18:45:17.973793 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:17.974807 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:17.976037 5332 hybrid_clock.cc:648] HybridClock initialized: now 1790534717975976 us; error 64 us; skew 500 ppm
I20260927 18:45:17.976336 5332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:17 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534717, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:17.979544 5332 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:45:17.981208 5332 webserver.cc:460] Webserver started at http://127.31.26.195:46545/ using document root <none> and password file <none>
I20260927 18:45:17.981938 5332 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:17.982084 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:17.982396 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:17.985406 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "4c64df96db7644d0b79a35893b4e8b1e"
format_stamp: "Formatted at 2026-09-27 18:45:17 on dist-test-slave-rfd5"
server_key: "bc0fd748225cc71182f73ea14dec9a15"
server_key_iv: "a75b46dbf4a368d3f23a0202174e5752"
server_key_version: "encryptionkey@0"
I20260927 18:45:17.986189 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "4c64df96db7644d0b79a35893b4e8b1e"
format_stamp: "Formatted at 2026-09-27 18:45:17 on dist-test-slave-rfd5"
server_key: "bc0fd748225cc71182f73ea14dec9a15"
server_key_iv: "a75b46dbf4a368d3f23a0202174e5752"
server_key_version: "encryptionkey@0"
I20260927 18:45:17.991652 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:45:17.995385 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:17.996908 5332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:17.997169 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "4c64df96db7644d0b79a35893b4e8b1e"
format_stamp: "Formatted at 2026-09-27 18:45:17 on dist-test-slave-rfd5"
server_key: "bc0fd748225cc71182f73ea14dec9a15"
server_key_iv: "a75b46dbf4a368d3f23a0202174e5752"
server_key_version: "encryptionkey@0"
I20260927 18:45:17.997414 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:18.020363 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:18.025733 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:18.026047 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:18.027170 5332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:18.028932 5332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:18.029054 5332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:18.029279 5332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:18.029374 5332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:18.059722 5192 heartbeater.cc:503] Master 127.31.26.254:34367 was elected leader, sending a full tablet report...
I20260927 18:45:18.066605 5332 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:46505
I20260927 18:45:18.066622 5459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:46505 every 8 connection(s)
I20260927 18:45:18.067920 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:18.076860 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5332
I20260927 18:45:18.077047 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:18.077695 31851 external_mini_cluster.cc:1428] Setting key 9625fd620876ed3ba8dd148b67c6b03f
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534717, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:18.096987 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.071668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:35441 (local address 127.31.26.254:34367)
0927 18:45:18.072134 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.072141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.073345 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 18:45:18.076298 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.076311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.076440 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.076569 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.079349 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.080488 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.083193 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.084143 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.084274 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.089045 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.089050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.089054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.089147 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.092856 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.093875 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.093883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.093890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.094055 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.094795 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.094798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.094801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.095187 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.095345 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.096330 (+ 985us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.096583 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":214,"threads_started":1}
I20260927 18:45:18.098513 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.072080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34367 (local address 127.31.26.195:35441)
0927 18:45:18.073132 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:18.073191 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:18.075887 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:18.077232 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:18.077241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:18.078223 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:18.079169 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:18.079200 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.080648 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.080654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:18.083042 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:18.083060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.084411 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.084421 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:18.084660 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.085565 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:18.085609 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:18.088760 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:18.093048 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:18.093065 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:18.093130 (+ 65us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:18.093712 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:18.094192 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:18.094197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:18.094200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:18.094543 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:18.095355 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:18.095367 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:18.095862 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 18:45:18.096236 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 18:45:18.096880 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":429,"threads_started":1}
I20260927 18:45:18.100206 5460 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34367
I20260927 18:45:18.100639 5460 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:18.101758 5460 heartbeater.cc:511] Master 127.31.26.254:34367 requested a full tablet report, sending...
I20260927 18:45:18.103652 5003 ts_manager.cc:194] Registered new tserver with Master: 4c64df96db7644d0b79a35893b4e8b1e (127.31.26.195:46505)
I20260927 18:45:18.104840 5003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:35441
I20260927 18:45:18.107568 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/test-admin.keytab.
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.138881 31851 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 18:45:18.144496 5468 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:18.141783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.31.26.193:40021)
0927 18:45:18.142497 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.142504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.142531 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:45:18.142591 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.142598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.142820 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.143007 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.144032 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34618: BlockingRecv error: recv got EOF from 127.0.0.1:34618 (error 108)
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":361,"threads_started":1}
W20260927 18:45:18.144658 31851 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.146422 31851 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.148495 31851 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:45:18.165978 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.150889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.31.26.193:40021)
0927 18:45:18.151086 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.151092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.151172 (+ 80us) server_negotiation.cc:407] Connection header received
0927 18:45:18.151401 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.151410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.151525 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.151650 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.152740 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.153873 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.155210 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.155866 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.155971 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.159915 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.159929 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.159945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.159985 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.162986 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.163769 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.163778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.163780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.163914 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.164323 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.164329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.164332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.164691 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.165045 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.165410 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.165668 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:45:18.172420 5468 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:18.170559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.31.26.193:40021)
0927 18:45:18.170796 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.170804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.171028 (+ 224us) server_negotiation.cc:407] Connection header received
0927 18:45:18.171238 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.171242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.171310 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.171417 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.172176 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34624: BlockingRecv error: recv got EOF from 127.0.0.1:34624 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 18:45:18.172940 31851 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc)
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.174521 31851 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.175859 31851 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:45:18.190057 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.177181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.31.26.193:40021)
0927 18:45:18.177382 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.177388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.177636 (+ 248us) server_negotiation.cc:407] Connection header received
0927 18:45:18.177802 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.177805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.177873 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.177952 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.178910 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.179598 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.180733 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.181239 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.181332 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.184912 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.184917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.184920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.184964 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.187574 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.188217 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.188224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.188227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.188356 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.188731 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.188738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.188741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.189005 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.189305 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.189569 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.189771 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 18:45:18.194365 5468 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:18.192616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.31.26.193:40021)
0927 18:45:18.192825 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.192839 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.193063 (+ 224us) server_negotiation.cc:407] Connection header received
0927 18:45:18.193278 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.193282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.193349 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.193449 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.194133 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34644: BlockingRecv error: recv got EOF from 127.0.0.1:34644 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 18:45:18.194615 31851 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534668899567-31851-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.196070 31851 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:18.197391 31851 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:45:18.211169 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.198626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.31.26.193:40021)
0927 18:45:18.198826 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.198841 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.199063 (+ 222us) server_negotiation.cc:407] Connection header received
0927 18:45:18.199170 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.199174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.199225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.199300 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.200160 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.200846 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.202146 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.202668 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.202762 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.206071 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.206076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.206079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.206109 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.208702 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.209389 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.209393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.209395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.209537 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.209850 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.209853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.209855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.210128 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.210374 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.210659 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.210869 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:45:18.212946 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5064
I20260927 18:45:18.224895 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.213702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.31.26.193:40021)
0927 18:45:18.213923 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.213932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.214199 (+ 267us) server_negotiation.cc:407] Connection header received
0927 18:45:18.214351 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.214354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.214407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.214481 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.215383 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.216045 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.217437 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.217875 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.217969 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.219934 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.219938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.219941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.219970 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.222540 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.223131 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.223135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.223137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.223258 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.223570 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.223572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.223574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.223810 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.224000 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.224288 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.224509 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:45:18.225518 5187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:18.308540 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5064
I20260927 18:45:18.325312 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5198
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:45:18.340622 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.326354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.31.26.194:37181)
0927 18:45:18.327050 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.327057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.327085 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:45:18.327161 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.327168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.327384 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.327536 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.328571 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.329815 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.331038 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.331625 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.331737 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.335445 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.335455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.335470 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.335502 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.338273 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.338898 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.338905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.338907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.339024 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.339350 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.339353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.339355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.339664 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.339875 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.340148 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.340344 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":393,"threads_started":1}
I20260927 18:45:18.341969 5321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:18.424609 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5198
I20260927 18:45:18.440488 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5332
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:45:18.457262 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:18.441567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.31.26.195:46505)
0927 18:45:18.442143 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:18.442150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:18.442181 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:45:18.442241 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:18.442248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:18.442492 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:18.442621 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:18.443684 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.444728 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.446012 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:18.446578 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:18.446679 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:18.450817 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:18.450832 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:18.450851 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:18.450887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:18.453898 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.454940 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.454955 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.454974 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.455153 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:18.455582 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:18.455585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:18.455587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:18.455979 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:18.456209 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:18.456535 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:45:18.456801 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":276,"threads_started":1}
I20260927 18:45:18.458602 5455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:18.542309 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5332
I20260927 18:45:18.558003 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 4971
I20260927 18:45:18.559216 5033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:18.661233 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 4971
2026-09-27T18:45:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4048 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[5488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[5488](info): set up 2 sockets
Sep 27 18:45:18 dist-test-slave-rfd5 krb5kdc[5488](info): commencing operation
krb5kdc: starting...
W20260927 18:45:19.762471 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:19 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab.
Entry for principal oryx/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab.
I20260927 18:45:19.982510 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42015
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab
--principal=oryx/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:20.318115 5501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:20.318502 5501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:20.318677 5501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:20.326447 5501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:20.326606 5501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:20.326716 5501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:20.326818 5501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:20.326893 5501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:20.337078 5501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:42015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab
--principal=oryx/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:20.338604 5501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:20.340013 5501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:20.352283 5506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:20.352224 5507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:20.352262 5509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:20.353207 5501 server_base.cc:1034] running on GCE node
I20260927 18:45:20.353977 5501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:20.355353 5501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:20.356554 5501 hybrid_clock.cc:648] HybridClock initialized: now 1790534720356517 us; error 36 us; skew 500 ppm
I20260927 18:45:20.356945 5501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:20 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534720, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:20.360167 5501 init.cc:377] Logged in from keytab as oryx/127.31.26.254@KRBTEST.COM (short username oryx)
I20260927 18:45:20.361750 5501 webserver.cc:460] Webserver started at http://127.31.26.254:35889/ using document root <none> and password file <none>
I20260927 18:45:20.362499 5501 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:20.362645 5501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:20.362953 5501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:20.366058 5501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "2bd5ba21bf314b60a28df0efa8261ff7"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "77724ed0ccb0e1cf4e6aaa3fd2e4ca33"
server_key_iv: "1a4d65b500a01bcb7998b9574d545699"
server_key_version: "encryptionkey@0"
I20260927 18:45:20.366851 5501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "2bd5ba21bf314b60a28df0efa8261ff7"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "77724ed0ccb0e1cf4e6aaa3fd2e4ca33"
server_key_iv: "1a4d65b500a01bcb7998b9574d545699"
server_key_version: "encryptionkey@0"
I20260927 18:45:20.372175 5501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:20.375726 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:20.377182 5501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:20.377363 5501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "2bd5ba21bf314b60a28df0efa8261ff7"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "77724ed0ccb0e1cf4e6aaa3fd2e4ca33"
server_key_iv: "1a4d65b500a01bcb7998b9574d545699"
server_key_version: "encryptionkey@0"
I20260927 18:45:20.377568 5501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:20.402787 5501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:20.406992 5501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:20.407320 5501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:20.426229 5501 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:42015
I20260927 18:45:20.426265 5567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:42015 every 8 connection(s)
I20260927 18:45:20.427670 5501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:20.431490 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:20.434000 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5501
I20260927 18:45:20.434211 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:20.434894 31851 external_mini_cluster.cc:1428] Setting key 5d5864fae69acbe564408015f8cee019
I20260927 18:45:20.445348 5568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Bootstrap starting.
I20260927 18:45:20.449357 5568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:20.450430 5568 log.cc:824] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:45:20 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:20.453662 5568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: No bootstrap required, opened a new log
I20260927 18:45:20.460470 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:20.437378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.31.26.254:42015)
0927 18:45:20.438279 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:20.438295 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:20.438760 (+ 465us) server_negotiation.cc:407] Connection header received
0927 18:45:20.440861 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:20.440896 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:20.441568 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:20.442062 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:20.443441 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:20.445504 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:20.447123 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:20.448143 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:20.448351 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:20.452463 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:20.452471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:20.452487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:20.452527 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:20.455908 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:20.456733 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:20.456747 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:20.456754 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:20.456946 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:20.457402 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:20.457408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:20.457413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:20.458014 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:20.458400 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:20.458851 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 18:45:20.459331 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":397,"threads_started":1}
I20260927 18:45:20.462059 5568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } }
I20260927 18:45:20.462598 5568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:20.462716 5568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd5ba21bf314b60a28df0efa8261ff7, State: Initialized, Role: FOLLOWER
I20260927 18:45:20.463423 5568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } }
I20260927 18:45:20.463706 5568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:20.463876 5568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:20.464059 5568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:20.466485 5568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } }
I20260927 18:45:20.467108 5568 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd5ba21bf314b60a28df0efa8261ff7; no voters:
I20260927 18:45:20.468019 5568 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:20.468132 5573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:20.468753 5573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 2bd5ba21bf314b60a28df0efa8261ff7, State: Running, Role: LEADER
I20260927 18:45:20.469376 5573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } }
I20260927 18:45:20.470388 5568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:20.472616 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } }
I20260927 18:45:20.472586 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd5ba21bf314b60a28df0efa8261ff7. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } }
I20260927 18:45:20.473654 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:20.477228 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:20.479421 5580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:20.485217 5580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:20.494866 5580 catalog_manager.cc:1347] Generated new cluster ID: 6824644ba5254436b842b4f7238e3b8b
I20260927 18:45:20.494997 5580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:20.527396 5580 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:20.528818 5580 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:20.535440 5580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Generated new TSK 0
I20260927 18:45:20.536500 5580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab.
Entry for principal oryx/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab.
I20260927 18:45:20.602210 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42015
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:20.944818 5594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:20.945201 5594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:20.945369 5594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:20.953220 5594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:20.953377 5594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:20.953563 5594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:20.964217 5594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:20.965709 5594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:20.967051 5594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:20.980757 5602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:20.980863 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:20.980882 5600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:20.981704 5594 server_base.cc:1034] running on GCE node
I20260927 18:45:20.982369 5594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:20.983350 5594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:20.984582 5594 hybrid_clock.cc:648] HybridClock initialized: now 1790534720984537 us; error 40 us; skew 500 ppm
I20260927 18:45:20.984977 5594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:20 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534720, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:20.988305 5594 init.cc:377] Logged in from keytab as oryx/127.31.26.193@KRBTEST.COM (short username oryx)
I20260927 18:45:20.989944 5594 webserver.cc:460] Webserver started at http://127.31.26.193:43815/ using document root <none> and password file <none>
I20260927 18:45:20.990696 5594 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:20.990854 5594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:20.991171 5594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:20.994320 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "d68fcb20df01479da5a3c24e529aef59"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "98c12c8febf2c454610acd47c0ac98df"
server_key_iv: "a02a6fa3f33ff77c3726f2b057133dfa"
server_key_version: "encryptionkey@0"
I20260927 18:45:20.995153 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "d68fcb20df01479da5a3c24e529aef59"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "98c12c8febf2c454610acd47c0ac98df"
server_key_iv: "a02a6fa3f33ff77c3726f2b057133dfa"
server_key_version: "encryptionkey@0"
I20260927 18:45:21.000731 5594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:21.004514 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:21.006201 5594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:21.006399 5594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "d68fcb20df01479da5a3c24e529aef59"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "98c12c8febf2c454610acd47c0ac98df"
server_key_iv: "a02a6fa3f33ff77c3726f2b057133dfa"
server_key_version: "encryptionkey@0"
I20260927 18:45:21.006630 5594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:21.031607 5594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:21.036590 5594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:21.036921 5594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:21.038022 5594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:21.039448 5594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:21.039556 5594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:21.039708 5594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:21.039781 5594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:21.074733 5594 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34509
I20260927 18:45:21.074834 5721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34509 every 8 connection(s)
I20260927 18:45:21.076129 5594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:21.077057 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5594
I20260927 18:45:21.077323 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:21.078137 31851 external_mini_cluster.cc:1428] Setting key b2eb06a5c1d8ee7e4b20e76dea86b2f5
Sep 27 18:45:21 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534720, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:45:21.100680 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:21.079319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:34357 (local address 127.31.26.254:42015)
0927 18:45:21.079818 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:21.079823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:21.081282 (+ 1459us) server_negotiation.cc:407] Connection header received
0927 18:45:21.083150 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:21.083159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:21.083253 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:21.083371 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:21.085021 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.086282 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.088519 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.089338 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.089496 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:21.093192 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:21.093202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:21.093211 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:21.093260 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:21.097263 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:21.098151 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:21.098159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:21.098163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:21.098322 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:21.098816 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:21.098823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:21.098828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:21.099233 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:21.099375 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:21.100189 (+ 814us) server_negotiation.cc:299] Negotiation successful
0927 18:45:21.100379 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":276,"threads_started":1}
I20260927 18:45:21.101958 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:21.080180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:34357)
0927 18:45:21.081107 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:21.081156 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:21.082869 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:21.083614 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:21.083624 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:21.084210 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:21.084838 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:21.084856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.086487 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.086494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:21.088335 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:21.088349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.089582 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.089587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:21.089811 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:21.090576 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:21.090600 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:21.092887 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:21.097451 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:21.097460 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:21.097476 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:21.097989 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:21.098467 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:21.098471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:21.098473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:21.098686 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:21.099370 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:21.099378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:21.099828 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 18:45:21.100079 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 18:45:21.100525 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.194@KRBTEST.COM" created.
I20260927 18:45:21.104023 5722 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42015
I20260927 18:45:21.104401 5722 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:21.105379 5722 heartbeater.cc:511] Master 127.31.26.254:42015 requested a full tablet report, sending...
I20260927 18:45:21.107982 5533 ts_manager.cc:194] Registered new tserver with Master: d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509)
I20260927 18:45:21.109647 5533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:34357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab.
Entry for principal oryx/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab.
I20260927 18:45:21.130875 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42015
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:21.477203 5728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:21.477527 5728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:21.477716 5728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:21.485589 5728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:21.485728 5728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:21.485934 5728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:21.496212 5728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:21.497680 5728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:21.499150 5728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:21.513623 5734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:21.513643 5736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:21.513702 5733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:21.514633 5728 server_base.cc:1034] running on GCE node
I20260927 18:45:21.515239 5728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:21.516181 5728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:21.517411 5728 hybrid_clock.cc:648] HybridClock initialized: now 1790534721517356 us; error 42 us; skew 500 ppm
I20260927 18:45:21.517745 5728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:21 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534721, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:21.529415 5728 init.cc:377] Logged in from keytab as oryx/127.31.26.194@KRBTEST.COM (short username oryx)
I20260927 18:45:21.531139 5728 webserver.cc:460] Webserver started at http://127.31.26.194:39361/ using document root <none> and password file <none>
I20260927 18:45:21.531961 5728 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:21.532125 5728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:21.532464 5728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:21.535564 5728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "b70f9613e05e49b79ea32bd28fe43ec8"
format_stamp: "Formatted at 2026-09-27 18:45:21 on dist-test-slave-rfd5"
server_key: "918a47cb5b4bc44a6acb1fb52724c88a"
server_key_iv: "3908aa296359f56cff686a2c124c4185"
server_key_version: "encryptionkey@0"
I20260927 18:45:21.536432 5728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "b70f9613e05e49b79ea32bd28fe43ec8"
format_stamp: "Formatted at 2026-09-27 18:45:21 on dist-test-slave-rfd5"
server_key: "918a47cb5b4bc44a6acb1fb52724c88a"
server_key_iv: "3908aa296359f56cff686a2c124c4185"
server_key_version: "encryptionkey@0"
I20260927 18:45:21.541930 5728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:45:21.545717 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:21.547169 5728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:45:21.547376 5728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "b70f9613e05e49b79ea32bd28fe43ec8"
format_stamp: "Formatted at 2026-09-27 18:45:21 on dist-test-slave-rfd5"
server_key: "918a47cb5b4bc44a6acb1fb52724c88a"
server_key_iv: "3908aa296359f56cff686a2c124c4185"
server_key_version: "encryptionkey@0"
I20260927 18:45:21.547602 5728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:21.571165 5728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:21.575932 5728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:21.576269 5728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:21.577392 5728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:21.578969 5728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:21.579077 5728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:21.579233 5728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:21.579305 5728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:21.613624 5728 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:43109
I20260927 18:45:21.613682 5855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:43109 every 8 connection(s)
I20260927 18:45:21.614940 5728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:21.623629 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5728
I20260927 18:45:21.623842 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:21.624285 31851 external_mini_cluster.cc:1428] Setting key bba06de17161ee6040e1359f0d0ee2a0
Sep 27 18:45:21 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534721, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:21.635177 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:21.617597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:57189 (local address 127.31.26.254:42015)
0927 18:45:21.618022 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:21.618026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:21.618911 (+ 885us) server_negotiation.cc:407] Connection header received
0927 18:45:21.620712 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:21.620717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:21.620810 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:21.620926 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:21.622466 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.623156 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.625041 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.625646 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.625751 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:21.629431 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:21.629436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:21.629440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:21.629471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:21.632059 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:21.632820 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:21.632826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:21.632834 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:21.632959 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:21.633440 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:21.633446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:21.633450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:21.633717 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:21.633886 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:21.634593 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 18:45:21.634810 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":237,"threads_started":1}
I20260927 18:45:21.636210 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:21.618066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.194:57189)
0927 18:45:21.618746 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:21.618794 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:21.620433 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:21.621209 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:21.621218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:21.621726 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:21.622289 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:21.622307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.623339 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.623343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:21.624878 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:21.624896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:21.625881 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:21.625888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:21.626216 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:21.626971 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:21.626995 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:21.629193 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:21.632218 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:21.632227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:21.632246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:21.632673 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:21.633068 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:21.633074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:21.633078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:21.633328 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:21.633861 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:21.633870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:21.634225 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 18:45:21.634489 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 18:45:21.635060 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
I20260927 18:45:21.637791 5856 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42015
I20260927 18:45:21.638142 5856 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:21.639067 5856 heartbeater.cc:511] Master 127.31.26.254:42015 requested a full tablet report, sending...
I20260927 18:45:21.640885 5533 ts_manager.cc:194] Registered new tserver with Master: b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109)
I20260927 18:45:21.642223 5533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:57189
WARNING: no policy specified for oryx/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab.
Entry for principal oryx/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab.
I20260927 18:45:21.672708 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42015
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:22.000798 5862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:22.001228 5862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:22.001389 5862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:22.008832 5862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:22.008962 5862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:22.009198 5862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:22.019699 5862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:22.021039 5862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:22.022437 5862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.036046 5870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.036077 5868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.036250 5867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.036919 5862 server_base.cc:1034] running on GCE node
I20260927 18:45:22.037644 5862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.038636 5862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.039881 5862 hybrid_clock.cc:648] HybridClock initialized: now 1790534722039841 us; error 34 us; skew 500 ppm
I20260927 18:45:22.040256 5862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.043562 5862 init.cc:377] Logged in from keytab as oryx/127.31.26.195@KRBTEST.COM (short username oryx)
I20260927 18:45:22.045135 5862 webserver.cc:460] Webserver started at http://127.31.26.195:39929/ using document root <none> and password file <none>
I20260927 18:45:22.045863 5862 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.046013 5862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.046329 5862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:22.049453 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "35f0bbe3162a4abe8a0b29b10489671c"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "bbda21c53d2699e1a91354a0832ad0d7"
server_key_iv: "23d95110e6d8196d11a5c8475470e54c"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.050290 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "35f0bbe3162a4abe8a0b29b10489671c"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "bbda21c53d2699e1a91354a0832ad0d7"
server_key_iv: "23d95110e6d8196d11a5c8475470e54c"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.056249 5862 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:45:22.060232 5877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.061951 5862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:22.062206 5862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "35f0bbe3162a4abe8a0b29b10489671c"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "bbda21c53d2699e1a91354a0832ad0d7"
server_key_iv: "23d95110e6d8196d11a5c8475470e54c"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.062435 5862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.075294 5862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.080124 5862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:22.080415 5862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.081409 5862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:22.082825 5862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:22.082927 5862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.083081 5862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:22.083158 5862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.113286 5722 heartbeater.cc:503] Master 127.31.26.254:42015 was elected leader, sending a full tablet report...
I20260927 18:45:22.121328 5862 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:41441
I20260927 18:45:22.121424 5989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:41441 every 8 connection(s)
I20260927 18:45:22.122861 5862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:22.123378 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5862
I20260927 18:45:22.123565 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:22.124015 31851 external_mini_cluster.cc:1428] Setting key 91f00bef170cb3cb83397e8aa900fafd
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:22.146310 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.127409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:37921 (local address 127.31.26.254:42015)
0927 18:45:22.127585 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.127590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.130468 (+ 2878us) server_negotiation.cc:407] Connection header received
0927 18:45:22.132356 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.132361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.132465 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.132583 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.134267 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.135013 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.136597 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.136996 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.137166 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.140903 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.140908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.140911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.140944 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.143531 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.144305 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.144309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.144311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.144426 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.144802 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.144805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.144807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.145063 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.145291 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.145889 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.146059 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:45:22.147365 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.126858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:37921)
0927 18:45:22.130298 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.130345 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.132065 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.132844 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.132853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.133478 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.134091 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.134111 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.135177 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.135187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.136439 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.136455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.137274 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.137281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.137521 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.138368 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.138391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.140702 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.143677 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.143684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.143698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.144169 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.144519 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.144523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.144525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.144701 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.145231 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.145238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.145556 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.145798 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.146203 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3168,"thread_start_us":2924,"threads_started":1}
I20260927 18:45:22.148831 5990 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42015
I20260927 18:45:22.149178 5990 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:22.149853 5990 heartbeater.cc:511] Master 127.31.26.254:42015 requested a full tablet report, sending...
I20260927 18:45:22.151414 5533 ts_manager.cc:194] Registered new tserver with Master: 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441)
I20260927 18:45:22.152376 5533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:37921
I20260927 18:45:22.152981 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.26.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.253.keytab.
Entry for principal oryx/127.31.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.253.keytab.
I20260927 18:45:22.201366 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:42695
--webserver_interface=127.31.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.253.keytab
--principal=oryx/127.31.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:42015,127.31.26.253:42695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:22.532274 5995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:22.532609 5995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:22.532783 5995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:22.540235 5995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:22.540357 5995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:22.540405 5995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:22.540485 5995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:22.540555 5995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:22.550635 5995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:42015,127.31.26.253:42695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.253.keytab
--principal=oryx/127.31.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
W20260927 18:45:22.551038 5995 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.26.254:42015,127.31.26.253:42695'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 18:45:22.552441 5995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:22.553956 5995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.565922 6000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.565889 6001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.566285 6003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.566694 5995 server_base.cc:1034] running on GCE node
I20260927 18:45:22.567528 5995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.568939 5995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.570180 5995 hybrid_clock.cc:648] HybridClock initialized: now 1790534722570124 us; error 43 us; skew 500 ppm
I20260927 18:45:22.570518 5995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.573994 5995 init.cc:377] Logged in from keytab as oryx/127.31.26.253@KRBTEST.COM (short username oryx)
I20260927 18:45:22.575659 5995 webserver.cc:460] Webserver started at http://127.31.26.253:34127/ using document root <none> and password file <none>
I20260927 18:45:22.576390 5995 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.576529 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.576849 5995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:22.580009 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/instance:
uuid: "debbbd3ea27044f6b6ef62e85d86cde6"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "223f701056ab02f02e67d06c4a49b734"
server_key_iv: "8965baeee4604e43afca2b3e1dac6634"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.580835 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal/instance:
uuid: "debbbd3ea27044f6b6ef62e85d86cde6"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "223f701056ab02f02e67d06c4a49b734"
server_key_iv: "8965baeee4604e43afca2b3e1dac6634"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.586174 5995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:45:22.589679 6010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.591116 5995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:45:22.591315 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
uuid: "debbbd3ea27044f6b6ef62e85d86cde6"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "223f701056ab02f02e67d06c4a49b734"
server_key_iv: "8965baeee4604e43afca2b3e1dac6634"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.591539 5995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.607955 5995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.612691 5995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:22.612982 5995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.632361 5995 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.253:42695
I20260927 18:45:22.632385 6061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.253:42695 every 8 connection(s)
I20260927 18:45:22.633927 5995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
I20260927 18:45:22.637552 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 5995
I20260927 18:45:22.637890 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:22.645709 5856 heartbeater.cc:503] Master 127.31.26.254:42015 was elected leader, sending a full tablet report...
I20260927 18:45:22.648514 6062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.253@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:22.665539 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.639932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.31.26.253:42695)
0927 18:45:22.640774 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.640786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.640835 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:45:22.642664 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.642689 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.643118 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.643521 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.645551 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.648087 (+ 2536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.649971 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.650692 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.650886 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.656710 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.656718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.656730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.656766 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.660702 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.661515 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.661524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.661535 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.661697 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.662223 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.662227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.662229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.662919 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.663216 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.663774 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.664229 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":401,"threads_started":1}
I20260927 18:45:22.667745 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.651770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34454)
0927 18:45:22.652800 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.652859 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.653043 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.653484 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.653492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.654401 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.654785 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.654801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.656150 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.656158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.657536 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.657548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.658236 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.658240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.658422 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.659059 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.659076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.661394 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.664969 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.664978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.664985 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.665588 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.666046 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.666056 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.666059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.666226 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.666831 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.666838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.667016 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.667180 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.667408 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":532,"threads_started":1}
I20260927 18:45:22.667898 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.651789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.31.26.254:42015)
0927 18:45:22.652384 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.652390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.652951 (+ 561us) server_negotiation.cc:407] Connection header received
0927 18:45:22.653211 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.653215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.653277 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.653372 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.654959 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.656000 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.657683 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.658080 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.658186 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.661618 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.661623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.661626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.661659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.664836 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.665732 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.665741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.665746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.665932 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.666367 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.666373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.666379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.666706 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.666817 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.667326 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.667561 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":296,"threads_started":1}
I20260927 18:45:22.669647 6062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.253@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
I20260927 18:45:22.682071 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.670199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53056)
0927 18:45:22.670400 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.670418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.670532 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.670797 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.670805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.671178 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.671327 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.671335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.672290 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.672293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.673239 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.673250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.673904 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.673908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.674025 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.674676 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.674691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.676898 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.679821 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.679825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.679828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.680164 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.680510 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.680513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.680515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.680659 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.681288 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.681292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.681413 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.681546 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.681836 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:45:22.682317 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.670433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.31.26.253:42695)
0927 18:45:22.670567 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.670571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.670585 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:45:22.670622 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.670625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.670675 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.670811 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.671440 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.672170 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.673372 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.673771 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.673876 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.677054 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.677059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.677062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.677160 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.679702 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.680259 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.680262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.680265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.680385 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.680770 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.680773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.680781 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.681175 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.681290 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.681640 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.681973 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":51}
I20260927 18:45:22.686901 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Bootstrap starting.
I20260927 18:45:22.690055 6062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:22.691053 6062 log.cc:824] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:22.693954 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: No bootstrap required, opened a new log
I20260927 18:45:22.701714 6062 raft_consensus.cc:348] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } }
I20260927 18:45:22.702128 6062 raft_consensus.cc:374] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:22.702217 6062 raft_consensus.cc:729] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debbbd3ea27044f6b6ef62e85d86cde6, State: Initialized, Role: FOLLOWER
I20260927 18:45:22.702790 6062 consensus_queue.cc:260] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } }
I20260927 18:45:22.703687 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } } }
I20260927 18:45:22.704010 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:45:22.704830 6062 sys_catalog.cc:564] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:45:22.713868 6080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:45:22.714053 6080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:45:22.717139 5995 master_runner.cc:428] Detected that this master debbbd3ea27044f6b6ef62e85d86cde6 is joining an existing cluster
I20260927 18:45:22.717260 5995 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.26.253:42695
I20260927 18:45:22.739665 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.721794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.31.26.254:42015)
0927 18:45:22.721997 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.722005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.722710 (+ 705us) server_negotiation.cc:407] Connection header received
0927 18:45:22.722981 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.722990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.723095 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.723171 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.724760 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.725946 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.728228 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.728963 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.729074 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.731939 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.731948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.731956 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.732004 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.735887 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.736885 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.736893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.736899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.737069 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.737633 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.737641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.737647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.738076 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.738210 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.738905 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.739196 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:45:22.740983 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.721832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34464)
0927 18:45:22.722600 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.722622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.722830 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.723413 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.723420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.724164 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.724587 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.724603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.726127 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.726134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.728040 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.728057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.729446 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.729454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.729679 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.730420 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.730448 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.731706 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.736080 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.736087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.736094 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.736732 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.737274 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.737282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.737289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.737494 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.738326 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.738334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.738582 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.739952 (+ 1370us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.740374 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":391,"threads_started":1}
WARNING: no policy specified for oryx/127.31.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.252@KRBTEST.COM" created.
I20260927 18:45:22.762324 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.723057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.31.26.253:42695)
0927 18:45:22.723231 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.723237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.735245 (+ 12008us) server_negotiation.cc:407] Connection header received
0927 18:45:22.735700 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.735706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.735787 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.735859 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.737487 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.738689 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.740984 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.741741 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.741848 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.745312 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.745318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.745322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.745359 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.749058 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.759623 (+ 10565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.759633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.759639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.759854 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.760376 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.760383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.760389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.760886 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.761104 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.761655 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.761919 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:45:22.763657 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.722742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53062)
0927 18:45:22.735117 (+ 12375us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.735141 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.735539 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.736063 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.736069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.736855 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.737306 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.737323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.738924 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.738930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.740800 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.740816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.742963 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.742969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.743176 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.743935 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.743960 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.745119 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.758509 (+ 13390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.758519 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.758527 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.759415 (+ 888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.760030 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.760037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.760043 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.760256 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.761186 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.761193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.761379 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.762911 (+ 1532us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.763178 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12214,"thread_start_us":12512,"threads_started":1}
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534720, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.254@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
I20260927 18:45:22.781850 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.749347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53078)
0927 18:45:22.762523 (+ 13176us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.762568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.762799 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.763413 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.763423 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.764378 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.764701 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.764718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.766729 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.766736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.768407 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.768431 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.769635 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.769642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.769867 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.770715 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.770744 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.773968 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.778506 (+ 4538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.778526 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.778537 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.779081 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.779552 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.779558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.779562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.779784 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.780553 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.780561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.780792 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.780999 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.781401 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12862,"thread_start_us":12720,"threads_started":1}
I20260927 18:45:22.781977 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.749465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.31.26.253:42695)
0927 18:45:22.761857 (+ 12392us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.761863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.762653 (+ 790us) server_negotiation.cc:407] Connection header received
0927 18:45:22.763088 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.763095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.763183 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.763324 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.764948 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.766549 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.768576 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.769453 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.769592 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.774252 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.774259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.774265 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.774311 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.778317 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.779267 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.779272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.779274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.779418 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.779900 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.779906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.779910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.780359 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.780501 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.781287 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.781527 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12245,"thread_start_us":12397,"threads_started":1}
I20260927 18:45:22.784371 5533 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.26.253:42695: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6"
member_type: NON_VOTER
last_known_addr {
host: "127.31.26.253"
port: 42695
}
attrs {
promote: true
}
}
dest_uuid: "2bd5ba21bf314b60a28df0efa8261ff7"
cas_config_opid_index: -1
I20260927 18:45:22.785507 5533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.252.keytab.
Entry for principal oryx/127.31.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.252.keytab.
I20260927 18:45:22.789351 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:39475
--webserver_interface=127.31.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.252.keytab
--principal=oryx/127.31.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:22.794126 6037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:22.793632 6094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER debbbd3ea27044f6b6ef62e85d86cde6 (127.31.26.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } }
I20260927 18:45:22.798326 6037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Refusing update from remote peer 2bd5ba21bf314b60a28df0efa8261ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 18:45:22.798731 6094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.26.253:42695
I20260927 18:45:22.799352 6094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:22.799769 6093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } }
I20260927 18:45:22.800338 6093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:22.811127 5995 master.cc:537] Master@127.31.26.253:42695 shutting down...
W20260927 18:45:22.817443 5519 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 18:45:22.826450 5995 raft_consensus.cc:2229] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:22.826913 5995 raft_consensus.cc:2258] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:22.827098 5995 tablet_replica.cc:330] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: stopping tablet replica
W20260927 18:45:22.830569 6011 connection.cc:441] server connection from 127.0.0.1:53078 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53078 (request call id 3) could send its response
W20260927 18:45:22.831599 5519 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 -> Peer debbbd3ea27044f6b6ef62e85d86cde6 (127.31.26.253:42695): Couldn't send request to peer debbbd3ea27044f6b6ef62e85d86cde6. Status: Network error: recv got EOF from 127.31.26.253:42695 (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:45:22.834983 5995 master.cc:552] Master@127.31.26.253:42695 shutdown complete.
I20260927 18:45:22.835070 5995 master_runner.cc:305] Clearing existing system tablet
I20260927 18:45:22.835482 5995 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.835569 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.844756 5995 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.005s
I20260927 18:45:22.846604 6100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.846995 5995 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.847292 5995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 18:45:22.847388 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
uuid: "debbbd3ea27044f6b6ef62e85d86cde6"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "223f701056ab02f02e67d06c4a49b734"
server_key_iv: "8965baeee4604e43afca2b3e1dac6634"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.852195 5995 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:22.859776 5995 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:45:22.859979 5995 log.cc:1196] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 18:45:22.860487 5995 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Deleting consensus metadata
I20260927 18:45:22.860765 5995 master_runner.cc:315] Copying system tablet from 127.31.26.254:42015
I20260927 18:45:22.867347 5995 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.26.254:42015
I20260927 18:45:22.888796 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.868516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.31.26.254:42015)
0927 18:45:22.868720 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:22.868725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:22.873780 (+ 5055us) server_negotiation.cc:407] Connection header received
0927 18:45:22.873992 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:22.873998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:22.874099 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:22.874222 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:22.875383 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.876516 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.878418 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.879166 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.879306 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.881755 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:22.881762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:22.881768 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:22.881832 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:22.885533 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.886441 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.886447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.886451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.886620 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:22.887078 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:22.887084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:22.887088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:22.887441 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:22.887593 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:22.888155 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 18:45:22.888400 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:45:22.888792 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:22.868311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34466)
0927 18:45:22.873613 (+ 5302us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:22.873640 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:22.873846 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:22.874273 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:22.874278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:22.874825 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:22.875128 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.875141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.876711 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.876716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.878221 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:22.878239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:22.879357 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:22.879363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:22.879546 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:22.880334 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:22.880361 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:22.881521 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:22.885711 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.885717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.885721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.886284 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.886757 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:22.886762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:22.886765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:22.886942 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:22.887579 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:22.887584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:22.887800 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 18:45:22.888018 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 18:45:22.888362 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5139,"thread_start_us":4969,"threads_started":1}
I20260927 18:45:22.890311 5553 tablet_copy_service.cc:140] P 2bd5ba21bf314b60a28df0efa8261ff7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer debbbd3ea27044f6b6ef62e85d86cde6 ({username='oryx', principal='oryx/127.31.26.253@KRBTEST.COM'} at 127.0.0.1:34466)
I20260927 18:45:22.890620 5553 tablet_copy_service.cc:161] P 2bd5ba21bf314b60a28df0efa8261ff7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer debbbd3ea27044f6b6ef62e85d86cde6 at {username='oryx', principal='oryx/127.31.26.253@KRBTEST.COM'} at 127.0.0.1:34466: session id = debbbd3ea27044f6b6ef62e85d86cde6-00000000000000000000000000000000
I20260927 18:45:22.895247 5553 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:45:22.897745 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:45:22.905400 5995 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: tablet copy: Starting download of 0 data blocks...
I20260927 18:45:22.905789 5995 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: tablet copy: Starting download of 1 WAL segments...
I20260927 18:45:22.908705 5995 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:45:22.913504 5553 tablet_copy_service.cc:342] P 2bd5ba21bf314b60a28df0efa8261ff7: Request end of tablet copy session debbbd3ea27044f6b6ef62e85d86cde6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.26.253@KRBTEST.COM'} at 127.0.0.1:34466
I20260927 18:45:22.913794 5553 tablet_copy_service.cc:433] P 2bd5ba21bf314b60a28df0efa8261ff7: ending tablet copy session debbbd3ea27044f6b6ef62e85d86cde6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer debbbd3ea27044f6b6ef62e85d86cde6
I20260927 18:45:22.925906 5995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:22.928872 6107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.929607 6108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:22.940766 6110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:22.941207 5995 server_base.cc:1034] running on GCE node
I20260927 18:45:22.941766 5995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:22.942436 5995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:22.943670 5995 hybrid_clock.cc:648] HybridClock initialized: now 1790534722943604 us; error 36 us; skew 500 ppm
I20260927 18:45:22.943882 5995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:22 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534722, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:22.946100 5995 init.cc:377] Logged in from keytab as oryx/127.31.26.253@KRBTEST.COM (short username oryx)
I20260927 18:45:22.947439 5995 webserver.cc:460] Webserver started at http://127.31.26.253:33909/ using document root <none> and password file <none>
I20260927 18:45:22.947811 5995 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:22.947889 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:22.951124 5995 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:22.953400 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:22.953850 5995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 18:45:22.953979 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
uuid: "debbbd3ea27044f6b6ef62e85d86cde6"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "223f701056ab02f02e67d06c4a49b734"
server_key_iv: "8965baeee4604e43afca2b3e1dac6634"
server_key_version: "encryptionkey@0"
I20260927 18:45:22.954147 5995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:22.971911 5995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:22.976101 5995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:22.996376 5995 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.253:42695
I20260927 18:45:22.996399 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.253:42695 every 8 connection(s)
I20260927 18:45:22.997413 5995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
I20260927 18:45:23.002575 6169 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:45:23.006510 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Bootstrap starting.
I20260927 18:45:23.025880 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:23.026467 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Bootstrap complete.
I20260927 18:45:23.027606 6169 raft_consensus.cc:348] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } }
I20260927 18:45:23.027889 6169 raft_consensus.cc:729] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: debbbd3ea27044f6b6ef62e85d86cde6, State: Initialized, Role: LEARNER
I20260927 18:45:23.028143 6169 consensus_queue.cc:260] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } }
I20260927 18:45:23.029125 6172 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } }
I20260927 18:45:23.029378 6169 sys_catalog.cc:564] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:23.029526 6172 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:45:23.037878 6183 catalog_manager.cc:1259] Loaded cluster ID: 6824644ba5254436b842b4f7238e3b8b
I20260927 18:45:23.038043 6183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: loading cluster ID for follower catalog manager: success
I20260927 18:45:23.042479 6183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: acquiring CA information for follower catalog manager: success
I20260927 18:45:23.045308 6183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:45:23.155110 5990 heartbeater.cc:503] Master 127.31.26.254:42015 was elected leader, sending a full tablet report...
W20260927 18:45:23.217741 6095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:23.218048 6095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:23.218145 6095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:23.225746 6095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:23.225873 6095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:23.225919 6095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:23.225978 6095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:23.226023 6095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:23.237012 6095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.252.keytab
--principal=oryx/127.31.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:23.238859 6095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:23.240211 6095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:23.252471 6189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:23.252519 6188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:23.252554 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:23.253345 6095 server_base.cc:1034] running on GCE node
I20260927 18:45:23.254313 6095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:23.256068 6095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:23.257297 6095 hybrid_clock.cc:648] HybridClock initialized: now 1790534723257237 us; error 53 us; skew 500 ppm
I20260927 18:45:23.258224 6095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534723, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:23.262012 6095 init.cc:377] Logged in from keytab as oryx/127.31.26.252@KRBTEST.COM (short username oryx)
I20260927 18:45:23.264019 6095 webserver.cc:460] Webserver started at http://127.31.26.252:35039/ using document root <none> and password file <none>
I20260927 18:45:23.264922 6095 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:23.265070 6095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:23.265443 6095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:23.266808 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.259289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53084)
0927 18:45:23.259522 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.259544 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.259699 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.260398 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.260404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.260435 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:23.260625 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.260650 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.262313 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.262323 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.264360 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.264381 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.265914 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.265923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.266152 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.266233 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.266355 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.266522 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:45:23.266817 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.259479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.31.26.253:42695)
0927 18:45:23.260060 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.260064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.260084 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:45:23.260144 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.260147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.260211 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.260318 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:23.260838 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.262061 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.264578 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.265660 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.265851 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.266053 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.266434 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.266544 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":343,"threads_started":1}
I20260927 18:45:23.267896 6144 raft_consensus.cc:1203] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 18:45:23.268576 6172 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd5ba21bf314b60a28df0efa8261ff7. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } }
I20260927 18:45:23.268843 6172 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:45:23.269459 6095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/instance:
uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
format_stamp: "Formatted at 2026-09-27 18:45:23 on dist-test-slave-rfd5"
server_key: "c6ed824c3986273ea998277c189ae9d3"
server_key_iv: "38ef57a16d1ddf286fd77898b03c974b"
server_key_version: "encryptionkey@0"
I20260927 18:45:23.270367 6095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal/instance:
uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
format_stamp: "Formatted at 2026-09-27 18:45:23 on dist-test-slave-rfd5"
server_key: "c6ed824c3986273ea998277c189ae9d3"
server_key_iv: "38ef57a16d1ddf286fd77898b03c974b"
server_key_version: "encryptionkey@0"
I20260927 18:45:23.276616 6095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 18:45:23.280809 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:23.282709 6095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:45:23.282925 6095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
format_stamp: "Formatted at 2026-09-27 18:45:23 on dist-test-slave-rfd5"
server_key: "c6ed824c3986273ea998277c189ae9d3"
server_key_iv: "38ef57a16d1ddf286fd77898b03c974b"
server_key_version: "encryptionkey@0"
I20260927 18:45:23.283150 6095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:23.313550 6095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:23.318022 6095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:23.318322 6095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:23.339165 6095 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.252:39475
I20260927 18:45:23.339205 6250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.252:39475 every 8 connection(s)
I20260927 18:45:23.340598 6095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
I20260927 18:45:23.344501 6251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:23.344895 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 6095
I20260927 18:45:23.354391 6251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.252@KRBTEST.COM
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534723, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.252@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:23.369815 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.347222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.31.26.252:39475)
0927 18:45:23.348045 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.348057 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.348097 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:45:23.350030 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.350054 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.350483 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.350896 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.352428 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.353866 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.355399 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.356059 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.356280 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.362116 (+ 5836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.362127 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.362141 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.362183 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.365403 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.366193 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.366205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.366217 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.366393 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.366842 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.366849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.366852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.367536 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.367830 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.368296 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.368731 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":359,"threads_started":1}
I20260927 18:45:23.372815 6255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.357010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34468)
0927 18:45:23.357888 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.357937 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.358099 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.358469 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.358477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.359312 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.360088 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.360104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.361126 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.361134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.362544 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.362555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.363264 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.363269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.363468 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.364230 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.364246 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.366449 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.369911 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.369920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.369926 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.370532 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.370959 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.370963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.370967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.371129 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.371909 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.371914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.372068 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.372261 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.372509 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":443,"threads_started":1}
I20260927 18:45:23.373128 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.357031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.31.26.254:42015)
0927 18:45:23.357294 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.357302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.358042 (+ 740us) server_negotiation.cc:407] Connection header received
0927 18:45:23.358235 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.358239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.358300 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.358382 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.360245 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.360947 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.362697 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.363075 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.363207 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.366661 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.366668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.366671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.366703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.369773 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.370690 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.370694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.370697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.370842 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.371314 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.371319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.371322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.371787 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.371945 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.372415 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.372643 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 18:45:23.374269 6251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534723, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.252@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
I20260927 18:45:23.388125 6255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.374888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53098)
0927 18:45:23.375515 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.375533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.375630 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.375947 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.375950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.376274 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.376409 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.376416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.377529 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.377533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.378489 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.378500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.379397 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.379402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.379529 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.380192 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.380209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.382420 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.385848 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.385852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.385854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.386254 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.386743 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.386747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.386751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.386885 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.387378 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.387381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.387501 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.387609 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.387905 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:45:23.388123 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.375121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.31.26.253:42695)
0927 18:45:23.375290 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.375294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.375653 (+ 359us) server_negotiation.cc:407] Connection header received
0927 18:45:23.375720 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.375724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.375796 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.375883 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.376571 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.377397 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.378625 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.379232 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.379349 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.382664 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.382670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.382674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.382714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.385690 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.386417 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.386422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.386425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.386627 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.386979 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.386982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.386984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.387286 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.387393 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.387719 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.387896 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:45:23.389233 6251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534723, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.252@KRBTEST.COM for oryx/127.31.26.252@KRBTEST.COM
I20260927 18:45:23.400980 6255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.389846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.0.0.1:39676)
0927 18:45:23.390004 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.390023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.390107 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.390379 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.390382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.390693 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.390825 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.390832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.391730 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.391733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.392553 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.392562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.393384 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.393389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.393519 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.394058 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.394074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.396048 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.398949 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.398953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.398956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.399302 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.399598 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.399601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.399604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.399739 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.400187 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.400191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.400331 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.400436 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.400626 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:45:23.401540 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.389909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.31.26.252:39475)
0927 18:45:23.390054 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.390058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.390072 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:45:23.390215 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.390218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.390270 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.390344 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.390937 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.391623 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.392726 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.393232 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.393350 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.396209 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.396213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.396217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.396244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.398820 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.399398 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.399402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.399404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.399510 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.399831 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.399834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.399836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.400095 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.400199 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.400516 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.400771 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":48}
I20260927 18:45:23.406139 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Bootstrap starting.
I20260927 18:45:23.409348 6251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:23.410395 6251 log.cc:824] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:23.413160 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: No bootstrap required, opened a new log
I20260927 18:45:23.420993 6251 raft_consensus.cc:348] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } }
I20260927 18:45:23.421520 6251 raft_consensus.cc:374] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:23.421653 6251 raft_consensus.cc:729] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a165b7bf2db84ddcbd3cb2b6482c55b1, State: Initialized, Role: FOLLOWER
I20260927 18:45:23.422254 6251 consensus_queue.cc:260] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } }
I20260927 18:45:23.423076 6257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } } }
I20260927 18:45:23.423364 6257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:45:23.424283 6251 sys_catalog.cc:564] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:45:23.434360 6268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:45:23.434593 6268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:45:23.438874 6095 master_runner.cc:428] Detected that this master a165b7bf2db84ddcbd3cb2b6482c55b1 is joining an existing cluster
I20260927 18:45:23.439025 6095 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.26.252:39475
I20260927 18:45:23.465955 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.446885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.31.26.254:42015)
0927 18:45:23.447107 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.447115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.447749 (+ 634us) server_negotiation.cc:407] Connection header received
0927 18:45:23.448086 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.448093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.448186 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.448262 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.449864 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.451542 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.454244 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.454979 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.455106 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.457971 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.457978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.457983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.458025 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.461994 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.463180 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.463185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.463189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.463348 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.463853 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.463858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.463862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.464256 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.464422 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.465077 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.465454 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:45:23.467114 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.446886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34478)
0927 18:45:23.447628 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.447649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.447904 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.448500 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.448506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.449299 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.449679 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.449693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.451840 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.451845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.454058 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.454072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.455463 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.455469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.455767 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.456526 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.456551 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.457746 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.462197 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.462202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.462207 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.463007 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.463500 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.463505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.463509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.463719 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.464517 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.464521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.464737 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.466241 (+ 1504us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.466613 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":32,"thread_start_us":387,"threads_started":1}
I20260927 18:45:23.467545 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.447901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.31.26.253:42695)
0927 18:45:23.448145 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.448154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.448766 (+ 612us) server_negotiation.cc:407] Connection header received
0927 18:45:23.449049 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.449055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.449162 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.449239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.451323 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.452669 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.456260 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.457171 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.457303 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.460203 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.460209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.460214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.460269 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.464200 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.465035 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.465044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.465050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.465266 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.465719 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.465727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.465732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.466058 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.466172 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.466890 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.467139 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:45:23.468783 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.447785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53108)
0927 18:45:23.448667 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.448683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.448877 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.449502 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.449512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.450446 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.451070 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.451087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.452914 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.452922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.456021 (+ 3099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.456041 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.457668 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.457678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.457928 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.458772 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.458803 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.459975 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.464378 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.464383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.464387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.464882 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.465407 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.465411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.465415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.465595 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.466279 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.466286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.466536 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.467951 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.468248 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":47,"thread_start_us":549,"threads_started":1}
I20260927 18:45:23.476629 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5594
I20260927 18:45:23.478057 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.448688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.0.0.1:39678)
0927 18:45:23.459371 (+ 10683us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.459392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.459722 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.460208 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.460213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.461354 (+ 1141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.461782 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.461798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.463343 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.463349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.465006 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.465017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.466698 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.466708 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.466894 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.468186 (+ 1292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.468234 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.469411 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.475004 (+ 5593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.475008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.475012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.475524 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.475937 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.475941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.475944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.476100 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.476832 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.476837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.477006 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.477198 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.477524 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10512,"thread_start_us":10790,"threads_started":1}
I20260927 18:45:23.479385 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.449152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.31.26.252:39475)
0927 18:45:23.449336 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.449341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.459469 (+ 10128us) server_negotiation.cc:407] Connection header received
0927 18:45:23.459869 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.459874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.459948 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.460013 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.461964 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.463122 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.465159 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.465931 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.466018 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.469618 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.469624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.469628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.469659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.474827 (+ 5168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.475655 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.475660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.475665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.475814 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.476230 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.476234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.476238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.476702 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.478440 (+ 1738us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.478561 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.478796 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534720, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.254@KRBTEST.COM for oryx/127.31.26.252@KRBTEST.COM
I20260927 18:45:23.491444 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.473630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.0.0.1:39680)
0927 18:45:23.473910 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.473930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.474036 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.474982 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.474988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.475449 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.475677 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.475689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.478083 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.478091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.479550 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.479567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.481060 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.481071 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.481363 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.482505 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.482523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.484619 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.488484 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.488490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.488495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.489014 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.489526 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.489531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.489534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.489702 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.490350 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.490356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.490543 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.490720 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.491034 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 18:45:23.491540 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.473759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.31.26.252:39475)
0927 18:45:23.474573 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.474579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.474602 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:45:23.474702 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.474708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.474804 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.475825 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.476149 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.477539 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.480049 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.480783 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.480987 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.484878 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.484887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.484890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.484926 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.488316 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.489214 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.489220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.489225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.489392 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.489816 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.489822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.489826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.490222 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.490370 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.490850 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.491079 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":503,"threads_started":1}
I20260927 18:45:23.493522 5533 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.26.252:39475: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
member_type: NON_VOTER
last_known_addr {
host: "127.31.26.252"
port: 39475
}
attrs {
promote: true
}
}
dest_uuid: "2bd5ba21bf314b60a28df0efa8261ff7"
cas_config_opid_index: 5
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.193@KRBTEST.COM
I20260927 18:45:23.494659 5533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } }
I20260927 18:45:23.496551 6093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.497169 6144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 LEARNER]: Refusing update from remote peer 2bd5ba21bf314b60a28df0efa8261ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:45:23.498488 6226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:23.498577 6279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 18:45:23.500356 6093 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.26.252:39475
I20260927 18:45:23.501080 6280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.501506 6280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:23.501992 6226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Refusing update from remote peer 2bd5ba21bf314b60a28df0efa8261ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:45:23.502091 6095 master.cc:537] Master@127.31.26.252:39475 shutting down...
I20260927 18:45:23.503526 6093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:45:23.513001 5519 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 -> Peer a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252:39475): Couldn't send request to peer a165b7bf2db84ddcbd3cb2b6482c55b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:45:23.513792 6144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.515851 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.478126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.31.26.193:34509)
0927 18:45:23.481685 (+ 3559us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.481695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.481727 (+ 32us) server_negotiation.cc:407] Connection header received
0927 18:45:23.481795 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.481805 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.482176 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.484376 (+ 2200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.484637 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.486266 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.488256 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.489317 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.489466 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.495645 (+ 6179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.495664 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.495683 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.495742 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.511661 (+ 15919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.512733 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.512745 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.512748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.512924 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.513510 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.513514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.513519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.513994 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.514306 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.515149 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.515419 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3362,"thread_start_us":3203,"threads_started":1}
I20260927 18:45:23.517277 5717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:23.518559 6172 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: NON_VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: true } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.518865 6172 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:45:23.519821 6279 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: attempting to promote NON_VOTER debbbd3ea27044f6b6ef62e85d86cde6 to VOTER
I20260927 18:45:23.520782 6279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } }
I20260927 18:45:23.523550 6144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 LEARNER]: Refusing update from remote peer 2bd5ba21bf314b60a28df0efa8261ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:45:23.524204 6279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 18:45:23.524696 5519 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 -> Peer a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252:39475): Couldn't send request to peer a165b7bf2db84ddcbd3cb2b6482c55b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:45:23.527053 6093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, debbbd3ea27044f6b6ef62e85d86cde6 (127.31.26.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.527732 6144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, debbbd3ea27044f6b6ef62e85d86cde6 (127.31.26.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.532276 6172 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.532547 6172 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:45:23.532956 6279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.533286 6279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:23.533617 6279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.533908 6279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:23.542649 6095 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:23.543123 6095 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:23.543325 6095 tablet_replica.cc:330] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: stopping tablet replica
I20260927 18:45:23.550340 6095 master.cc:552] Master@127.31.26.252:39475 shutdown complete.
I20260927 18:45:23.550439 6095 master_runner.cc:305] Clearing existing system tablet
I20260927 18:45:23.550848 6095 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:23.550940 6095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:23.554368 6095 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:23.556264 6286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:23.556684 6095 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:23.556954 6095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:45:23.557055 6095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
format_stamp: "Formatted at 2026-09-27 18:45:23 on dist-test-slave-rfd5"
server_key: "c6ed824c3986273ea998277c189ae9d3"
server_key_iv: "38ef57a16d1ddf286fd77898b03c974b"
server_key_version: "encryptionkey@0"
I20260927 18:45:23.562754 6095 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:23.569911 6095 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:45:23.570091 6095 log.cc:1196] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 18:45:23.570657 6095 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Deleting consensus metadata
I20260927 18:45:23.570946 6095 master_runner.cc:315] Copying system tablet from 127.31.26.254:42015
I20260927 18:45:23.573715 6095 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.26.254:42015
I20260927 18:45:23.593070 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.574631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.31.26.254:42015)
0927 18:45:23.574866 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.574873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.577720 (+ 2847us) server_negotiation.cc:407] Connection header received
0927 18:45:23.577917 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.577924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.578007 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.578120 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:23.579253 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.580450 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.582528 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.583180 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.583334 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.586257 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:23.586264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:23.586269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:23.586312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:23.589891 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.590833 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.590839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.590843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.591010 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:23.591482 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:23.591487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:23.591491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:23.591846 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:23.592000 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.592475 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.592721 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 18:45:23.593245 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.574562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34494)
0927 18:45:23.577558 (+ 2996us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.577581 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.577775 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.578169 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.578174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.578765 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:23.579051 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.579063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.580637 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.580643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.582330 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.582347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.583385 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.583391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.583607 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.584358 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:23.584383 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:23.585994 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:23.590065 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.590072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.590076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.590652 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.591144 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:23.591149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:23.591153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:23.591343 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:23.591977 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:23.591983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:23.592187 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.592348 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.592694 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2839,"thread_start_us":2713,"threads_started":1}
I20260927 18:45:23.594179 5553 tablet_copy_service.cc:140] P 2bd5ba21bf314b60a28df0efa8261ff7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a165b7bf2db84ddcbd3cb2b6482c55b1 ({username='oryx', principal='oryx/127.31.26.252@KRBTEST.COM'} at 127.0.0.1:34494)
I20260927 18:45:23.594385 5553 tablet_copy_service.cc:161] P 2bd5ba21bf314b60a28df0efa8261ff7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a165b7bf2db84ddcbd3cb2b6482c55b1 at {username='oryx', principal='oryx/127.31.26.252@KRBTEST.COM'} at 127.0.0.1:34494: session id = a165b7bf2db84ddcbd3cb2b6482c55b1-00000000000000000000000000000000
I20260927 18:45:23.598080 5553 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:45:23.600167 6095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:45:23.607304 6095 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: tablet copy: Starting download of 0 data blocks...
I20260927 18:45:23.607638 6095 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: tablet copy: Starting download of 1 WAL segments...
I20260927 18:45:23.610316 6095 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:45:23.616961 5553 tablet_copy_service.cc:342] P 2bd5ba21bf314b60a28df0efa8261ff7: Request end of tablet copy session a165b7bf2db84ddcbd3cb2b6482c55b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.26.252@KRBTEST.COM'} at 127.0.0.1:34494
I20260927 18:45:23.617143 5553 tablet_copy_service.cc:433] P 2bd5ba21bf314b60a28df0efa8261ff7: ending tablet copy session a165b7bf2db84ddcbd3cb2b6482c55b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a165b7bf2db84ddcbd3cb2b6482c55b1
I20260927 18:45:23.629467 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5594
I20260927 18:45:23.629649 6095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:23.638921 6293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:23.639190 6294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:23.654158 6296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:23.654457 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:34509
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:42015
--webserver_port=43815
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:23.654197 6095 server_base.cc:1034] running on GCE node
I20260927 18:45:23.655140 6095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:23.655809 6095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:23.657006 6095 hybrid_clock.cc:648] HybridClock initialized: now 1790534723656996 us; error 84 us; skew 500 ppm
I20260927 18:45:23.657284 6095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:23 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534723, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:23.659559 6095 init.cc:377] Logged in from keytab as oryx/127.31.26.252@KRBTEST.COM (short username oryx)
I20260927 18:45:23.661270 6095 webserver.cc:460] Webserver started at http://127.31.26.252:39145/ using document root <none> and password file <none>
I20260927 18:45:23.661614 6095 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:23.661734 6095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:23.665326 6095 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:23.667188 6304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:23.667615 6095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:45:23.667747 6095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
format_stamp: "Formatted at 2026-09-27 18:45:23 on dist-test-slave-rfd5"
server_key: "c6ed824c3986273ea998277c189ae9d3"
server_key_iv: "38ef57a16d1ddf286fd77898b03c974b"
server_key_version: "encryptionkey@0"
I20260927 18:45:23.667937 6095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:23.685910 6095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:23.689316 6095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:23.718119 6095 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.252:39475
I20260927 18:45:23.718152 6355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.252:39475 every 8 connection(s)
I20260927 18:45:23.719048 6095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
I20260927 18:45:23.726118 6356 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:45:23.731545 6356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Bootstrap starting.
I20260927 18:45:23.747112 6356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:23.747620 6356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: Bootstrap complete.
I20260927 18:45:23.748514 6356 raft_consensus.cc:348] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } }
I20260927 18:45:23.748775 6356 raft_consensus.cc:729] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a165b7bf2db84ddcbd3cb2b6482c55b1, State: Initialized, Role: LEARNER
I20260927 18:45:23.749006 6356 consensus_queue.cc:260] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } }
I20260927 18:45:23.751926 6359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.752306 6356 sys_catalog.cc:564] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:23.752456 6359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:45:23.771306 6370 catalog_manager.cc:1259] Loaded cluster ID: 6824644ba5254436b842b4f7238e3b8b
I20260927 18:45:23.771431 6370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: loading cluster ID for follower catalog manager: success
I20260927 18:45:23.773994 6370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: acquiring CA information for follower catalog manager: success
I20260927 18:45:23.775838 6370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:45:23.917001 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.908314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.0.0.1:39688)
0927 18:45:23.908596 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:23.908617 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:23.908754 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:23.909723 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:23.909729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:23.909766 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:23.909954 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.909967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.912006 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.912013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.914044 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:23.914061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.916179 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.916186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:23.916391 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.916431 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:45:23.916549 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:45:23.916714 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260927 18:45:23.917261 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:23.908553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.31.26.252:39475)
0927 18:45:23.909319 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:23.909325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:23.909349 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:45:23.909423 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:23.909428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:23.909515 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:23.909648 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:23.910119 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.911814 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.914250 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:23.915968 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:23.916136 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:23.916453 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:23.916591 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:45:23.916779 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":435,"threads_started":1}
I20260927 18:45:23.918496 6331 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 18:45:23.919229 6359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd5ba21bf314b60a28df0efa8261ff7. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:23.919502 6359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: This master's current role is: LEARNER
W20260927 18:45:24.079838 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:24.080199 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:24.080368 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:24.088428 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:24.088615 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:24.088820 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:24.099208 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:34509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=43815
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:24.100631 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:24.102084 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:24.119314 6376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:24.119393 6377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:24.119304 6379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:24.125454 6301 server_base.cc:1034] running on GCE node
I20260927 18:45:24.126170 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:24.127192 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:24.128428 6301 hybrid_clock.cc:648] HybridClock initialized: now 1790534724128383 us; error 43 us; skew 500 ppm
I20260927 18:45:24.128788 6301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:24.132190 6301 init.cc:377] Logged in from keytab as oryx/127.31.26.193@KRBTEST.COM (short username oryx)
I20260927 18:45:24.133989 6301 webserver.cc:460] Webserver started at http://127.31.26.193:43815/ using document root <none> and password file <none>
I20260927 18:45:24.134717 6301 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:24.134840 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:24.140241 6301 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:45:24.143589 6386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:24.145560 6301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:45:24.145709 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "d68fcb20df01479da5a3c24e529aef59"
format_stamp: "Formatted at 2026-09-27 18:45:20 on dist-test-slave-rfd5"
server_key: "98c12c8febf2c454610acd47c0ac98df"
server_key_iv: "a02a6fa3f33ff77c3726f2b057133dfa"
server_key_version: "encryptionkey@0"
I20260927 18:45:24.146742 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:24.173674 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:24.179023 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:24.179299 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:24.180334 6301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:24.182367 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:24.182459 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:24.182544 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:24.182597 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:24.220299 6301 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34509
I20260927 18:45:24.220397 6498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34509 every 8 connection(s)
I20260927 18:45:24.221932 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:24.225150 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 6301
I20260927 18:45:24.225559 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5728
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.252@KRBTEST.COM
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.194@KRBTEST.COM
I20260927 18:45:24.264110 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.224919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:60289 (local address 127.31.26.254:42015)
0927 18:45:24.225450 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.225458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.229150 (+ 3692us) server_negotiation.cc:407] Connection header received
0927 18:45:24.230659 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.230669 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.230794 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.230897 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.236839 (+ 5942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.238448 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.249284 (+ 10836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.249969 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.250134 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.256523 (+ 6389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.256533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.256538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.256619 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.260429 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.261393 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.261397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.261400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.261534 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.262014 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.262020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.262022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.262304 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.262467 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.263491 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.263731 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":270,"threads_started":1}
I20260927 18:45:24.265179 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.225712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:60289)
0927 18:45:24.226595 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:24.226658 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:24.230201 (+ 3543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:24.231695 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:24.231713 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:24.232731 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:24.233884 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.233920 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.238749 (+ 4829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.238758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.241742 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.241768 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.250517 (+ 8749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.250524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.251250 (+ 726us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.252582 (+ 1332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:24.252605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:24.256135 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:24.260617 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.260628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.260650 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.261241 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.261661 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.261666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.261670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.261873 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.262444 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:24.262453 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:24.262970 (+ 517us) client_negotiation.cc:769] Sending connection context
0927 18:45:24.263327 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 18:45:24.263833 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":338,"threads_started":1}
I20260927 18:45:24.267261 6500 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42015
I20260927 18:45:24.267614 6500 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:24.268239 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.227643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:44125 (local address 127.31.26.253:42695)
0927 18:45:24.234552 (+ 6909us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.234561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.235466 (+ 905us) server_negotiation.cc:407] Connection header received
0927 18:45:24.235550 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.235558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.235903 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.236506 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.243617 (+ 7111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.245192 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.249470 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.250259 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.250400 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.257620 (+ 7220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.257629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.257637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.257687 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.263918 (+ 6231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.265608 (+ 1690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.265617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.265623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.265800 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.266271 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.266277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.266280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.266698 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.266930 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.267642 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.267879 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6734,"thread_start_us":6568,"threads_started":1}
I20260927 18:45:24.268597 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.228513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.31.26.193:44125)
0927 18:45:24.235141 (+ 6628us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:24.235167 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:24.235334 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:24.241998 (+ 6664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:24.242006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:24.242906 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:24.243355 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.243371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.245427 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.245436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.249292 (+ 3856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.249303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.251263 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.251272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.251489 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.252338 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:24.252378 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:24.257356 (+ 4978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:24.264092 (+ 6736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.264099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.264104 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.264706 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.265936 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.265941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.265944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.266115 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.266998 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:24.267003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:24.267174 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 18:45:24.267363 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 18:45:24.268004 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"mutex_wait_us":288,"thread_start_us":6239,"threads_started":1}
I20260927 18:45:24.268664 6500 heartbeater.cc:511] Master 127.31.26.254:42015 requested a full tablet report, sending...
I20260927 18:45:24.269572 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.235523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:54289 (local address 127.31.26.252:39475)
0927 18:45:24.235761 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.235770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.244559 (+ 8789us) server_negotiation.cc:407] Connection header received
0927 18:45:24.245198 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.245207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.245317 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.245406 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.247492 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.248609 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.250385 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.251024 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.251119 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.257395 (+ 6276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.257402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.257406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.257449 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.262425 (+ 4976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.266005 (+ 3580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.266011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.266014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.266165 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.266715 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.266726 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.266733 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.267109 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.267253 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.268941 (+ 1688us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.269234 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:45:24.270380 6501 heartbeater.cc:347] Connected to a master server at 127.31.26.253:42695
I20260927 18:45:24.270467 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.235763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.31.26.193:54289)
0927 18:45:24.244397 (+ 8634us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:24.244428 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:24.244965 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:24.245693 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:24.245702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:24.246583 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:24.247203 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.247225 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.248777 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.248783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.250208 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.250224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.251398 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.251405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.251610 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.252438 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:24.252470 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:24.257154 (+ 4684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:24.264901 (+ 7747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.264908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.264911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.265461 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.266321 (+ 860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.266325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.266386 (+ 61us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.266575 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.267336 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:24.267340 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:24.267588 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 18:45:24.269743 (+ 2155us) client_negotiation.cc:240] Negotiation successful
0927 18:45:24.270051 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8439,"mutex_wait_us":78,"thread_start_us":8815,"threads_started":1}
I20260927 18:45:24.270519 6501 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:24.270876 5533 ts_manager.cc:194] Re-registered known tserver with Master: d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509)
I20260927 18:45:24.270975 6501 heartbeater.cc:511] Master 127.31.26.253:42695 requested a full tablet report, sending...
I20260927 18:45:24.271940 5533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:60289
I20260927 18:45:24.272892 6134 ts_manager.cc:194] Registered new tserver with Master: d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509)
I20260927 18:45:24.272974 6499 heartbeater.cc:347] Connected to a master server at 127.31.26.252:39475
I20260927 18:45:24.273141 6499 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:24.273573 6499 heartbeater.cc:511] Master 127.31.26.252:39475 requested a full tablet report, sending...
I20260927 18:45:24.275130 6321 ts_manager.cc:194] Registered new tserver with Master: d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509)
I20260927 18:45:24.276230 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.226479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.31.26.194:43109)
0927 18:45:24.248845 (+ 22366us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.248853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.248878 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:45:24.248930 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.248936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.249291 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.249452 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.252656 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.254394 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.256498 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.257512 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.257658 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.265710 (+ 8052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.265731 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.265749 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.265793 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.270094 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.271156 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.271166 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.271170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.271376 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.271844 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.271851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.271858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.272267 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.272948 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.274203 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.275908 (+ 1705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22172,"thread_start_us":22026,"threads_started":1}
I20260927 18:45:24.277521 5851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:24.362283 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5728
I20260927 18:45:24.379310 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:43109
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:42015
--webserver_port=39361
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:24.382848 6093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: NON_VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: true } } }
I20260927 18:45:24.383224 6093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:24.384529 6512 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: attempting to promote NON_VOTER a165b7bf2db84ddcbd3cb2b6482c55b1 to VOTER
I20260927 18:45:24.385253 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } }
I20260927 18:45:24.387204 6144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Refusing update from remote peer 2bd5ba21bf314b60a28df0efa8261ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 18:45:24.387248 6331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 LEARNER]: Refusing update from remote peer 2bd5ba21bf314b60a28df0efa8261ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 18:45:24.387764 6093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:45:24.388124 6511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:45:24.390617 6511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.392762 6331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.394007 6144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.395046 6511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.395347 6511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:24.395632 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.395910 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:24.396346 6511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.396615 6511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:24.398554 6359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.398895 6359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:45:24.407385 6517 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:24.407915 6517 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:45:24.732059 6514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:24.732543 6514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:24.732736 6514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:24.740979 6514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:24.741199 6514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:24.741403 6514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:24.751389 6514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:43109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=39361
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:24.752812 6514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:24.754261 6514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:24.768877 6522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:24.768927 6523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:24.769631 6525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:24.770087 6514 server_base.cc:1034] running on GCE node
I20260927 18:45:24.770783 6514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:24.771878 6514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:24.773125 6514 hybrid_clock.cc:648] HybridClock initialized: now 1790534724773037 us; error 67 us; skew 500 ppm
I20260927 18:45:24.773439 6514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:24.776729 6514 init.cc:377] Logged in from keytab as oryx/127.31.26.194@KRBTEST.COM (short username oryx)
I20260927 18:45:24.778354 6514 webserver.cc:460] Webserver started at http://127.31.26.194:39361/ using document root <none> and password file <none>
I20260927 18:45:24.779090 6514 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:24.779255 6514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:24.785439 6514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:45:24.788270 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:24.789795 6514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:24.789987 6514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "b70f9613e05e49b79ea32bd28fe43ec8"
format_stamp: "Formatted at 2026-09-27 18:45:21 on dist-test-slave-rfd5"
server_key: "918a47cb5b4bc44a6acb1fb52724c88a"
server_key_iv: "3908aa296359f56cff686a2c124c4185"
server_key_version: "encryptionkey@0"
I20260927 18:45:24.791008 6514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:24.805662 6514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:24.810643 6514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:24.810958 6514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:24.811987 6514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:24.813474 6514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:24.813601 6514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:24.813727 6514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:24.813846 6514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:24.849200 6514 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:43109
I20260927 18:45:24.849336 6644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:43109 every 8 connection(s)
I20260927 18:45:24.850868 6514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:24.855020 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 6514
I20260927 18:45:24.855377 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5862
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.195@KRBTEST.COM
I20260927 18:45:24.882544 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.856946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:43251 (local address 127.31.26.253:42695)
0927 18:45:24.857469 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.857478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.861981 (+ 4503us) server_negotiation.cc:407] Connection header received
0927 18:45:24.862050 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.862058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.862143 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.862253 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.863812 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.864942 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.867758 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.868417 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.868536 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.874489 (+ 5953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.874497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.874502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.874558 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.878172 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.879370 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.879377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.879381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.879539 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.880172 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.880178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.880181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.880583 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.880732 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.881880 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.882120 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":242,"threads_started":1}
I20260927 18:45:24.883289 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.854262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:40309 (local address 127.31.26.254:42015)
0927 18:45:24.854727 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.854731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.856067 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 18:45:24.860246 (+ 4179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.860256 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.860402 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.860551 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.863657 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.865244 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.867820 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.868672 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.868794 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.874168 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.874174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.874177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.874212 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.878924 (+ 4712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.880093 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.880100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.880105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.880260 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.880804 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.880809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.880812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.881283 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.881450 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.882209 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.882489 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":267,"threads_started":1}
I20260927 18:45:24.885140 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.854978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.194:40309)
0927 18:45:24.855873 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:24.855929 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:24.859778 (+ 3849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:24.861172 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:24.861195 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:24.862127 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:24.863349 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.863390 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.865507 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.865515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.867624 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.867642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.869159 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.869168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.869455 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.870235 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:24.870265 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:24.873905 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:24.879165 (+ 5260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.879173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.879180 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.879898 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.880413 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.880420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.880427 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.880657 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.881553 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:24.881560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:24.881860 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 18:45:24.883543 (+ 1683us) client_negotiation.cc:240] Negotiation successful
0927 18:45:24.883875 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":5,"thread_start_us":359,"threads_started":1}
I20260927 18:45:24.885174 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.856576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.31.26.194:43251)
0927 18:45:24.861718 (+ 5142us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:24.861739 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:24.861869 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:24.862497 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:24.862504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:24.863182 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:24.863627 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.863643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.865176 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.865183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.867596 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.867609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.868858 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.868865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.869220 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.870057 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:24.870099 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:24.874274 (+ 4175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:24.878383 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.878395 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.878421 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.879198 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.879715 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.879722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.879728 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.880019 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.880831 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:24.880841 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:24.881389 (+ 548us) client_negotiation.cc:769] Sending connection context
0927 18:45:24.882800 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0927 18:45:24.883417 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4955,"mutex_wait_us":76,"thread_start_us":4724,"threads_started":1}
I20260927 18:45:24.887013 6647 heartbeater.cc:347] Connected to a master server at 127.31.26.253:42695
I20260927 18:45:24.887141 6646 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42015
I20260927 18:45:24.887396 6646 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:24.887400 6647 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:24.888507 6646 heartbeater.cc:511] Master 127.31.26.254:42015 requested a full tablet report, sending...
I20260927 18:45:24.888530 6647 heartbeater.cc:511] Master 127.31.26.253:42695 requested a full tablet report, sending...
I20260927 18:45:24.888612 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.856822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.31.26.195:41441)
0927 18:45:24.860835 (+ 4013us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.860842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.860867 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:45:24.860922 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.860927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.861229 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.863717 (+ 2488us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.863985 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.865466 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.867429 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.868396 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.874121 (+ 5725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.876830 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.876851 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.876872 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.876925 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.881219 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.882469 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.882478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.882482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.883023 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.883747 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.883752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.883756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.884204 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.886700 (+ 2496us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.886886 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.888177 (+ 1291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3843,"thread_start_us":3685,"threads_started":1}
I20260927 18:45:24.890519 5985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:24.890746 6134 ts_manager.cc:194] Registered new tserver with Master: b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109)
I20260927 18:45:24.891184 5533 ts_manager.cc:194] Re-registered known tserver with Master: b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109)
I20260927 18:45:24.892408 5533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:40309
Sep 27 18:45:24 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534724, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.252@KRBTEST.COM
I20260927 18:45:24.901296 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.861953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:44657 (local address 127.31.26.252:39475)
0927 18:45:24.883104 (+ 21151us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:24.883112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:24.883132 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:45:24.883192 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:24.883197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:24.883280 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:24.883393 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:24.884924 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.886240 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.888305 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.888946 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.889073 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.894542 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:24.894551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:24.894556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:24.894608 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:24.898210 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.899144 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.899150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.899153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.899367 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:24.899843 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:24.899846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:24.899848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:24.900155 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:24.900320 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:24.900756 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:45:24.900928 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20621,"thread_start_us":22818,"threads_started":1}
I20260927 18:45:24.901507 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:24.866683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.31.26.194:44657)
0927 18:45:24.880387 (+ 13704us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:24.880408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:24.880517 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:24.883621 (+ 3104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:24.883625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:24.884342 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:24.884652 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.884664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.886401 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.886409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.888106 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:24.888123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:24.889379 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:24.889385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:24.889624 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:24.890294 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:24.890316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:24.894328 (+ 4012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:24.898385 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.898392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.898397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.898986 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.899516 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:24.899521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:24.899524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:24.899733 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:24.900287 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:24.900293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:24.900495 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 18:45:24.900660 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 18:45:24.901039 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13532,"thread_start_us":13899,"threads_started":1}
I20260927 18:45:24.902550 6645 heartbeater.cc:347] Connected to a master server at 127.31.26.252:39475
I20260927 18:45:24.902737 6645 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:24.903227 6645 heartbeater.cc:511] Master 127.31.26.252:39475 requested a full tablet report, sending...
I20260927 18:45:24.904433 6321 ts_manager.cc:194] Registered new tserver with Master: b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109)
I20260927 18:45:24.978334 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5862
I20260927 18:45:24.995215 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:41441
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:42015
--webserver_port=39929
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:25.276388 6500 heartbeater.cc:503] Master 127.31.26.254:42015 was elected leader, sending a full tablet report...
W20260927 18:45:25.342195 6658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:25.342540 6658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:25.342638 6658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:25.350103 6658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:25.350226 6658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:25.350399 6658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:25.361611 6658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:41441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=39929
--tserver_master_addrs=127.31.26.254:42015,127.31.26.253:42695,127.31.26.252:39475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:25.363009 6658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:25.364370 6658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:25.378391 6663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:25.378353 6666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:25.378419 6664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:25.379504 6658 server_base.cc:1034] running on GCE node
I20260927 18:45:25.380179 6658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:25.381158 6658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:25.382402 6658 hybrid_clock.cc:648] HybridClock initialized: now 1790534725382340 us; error 74 us; skew 500 ppm
I20260927 18:45:25.382784 6658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:25 dist-test-slave-rfd5 krb5kdc[5488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534725, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:25.386348 6658 init.cc:377] Logged in from keytab as oryx/127.31.26.195@KRBTEST.COM (short username oryx)
I20260927 18:45:25.387988 6658 webserver.cc:460] Webserver started at http://127.31.26.195:39929/ using document root <none> and password file <none>
I20260927 18:45:25.388744 6658 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:25.388906 6658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:25.394757 6658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 18:45:25.397975 6673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:25.399570 6658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:25.399772 6658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "35f0bbe3162a4abe8a0b29b10489671c"
format_stamp: "Formatted at 2026-09-27 18:45:22 on dist-test-slave-rfd5"
server_key: "bbda21c53d2699e1a91354a0832ad0d7"
server_key_iv: "23d95110e6d8196d11a5c8475470e54c"
server_key_version: "encryptionkey@0"
I20260927 18:45:25.400815 6658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:25.413566 6658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:25.418462 6658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:25.418776 6658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:25.419853 6658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:25.421377 6658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:25.421496 6658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:25.421634 6658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:25.421723 6658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:25.456100 6658 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:41441
I20260927 18:45:25.456187 6785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:41441 every 8 connection(s)
I20260927 18:45:25.457695 6658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:25.460703 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 6658
Sep 27 18:45:25 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534725, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.252@KRBTEST.COM
Sep 27 18:45:25 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534725, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.253@KRBTEST.COM
I20260927 18:45:25.496915 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.463007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:36097 (local address 127.31.26.252:39475)
0927 18:45:25.468162 (+ 5155us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.468172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.469413 (+ 1241us) server_negotiation.cc:407] Connection header received
0927 18:45:25.469666 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.469675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.469810 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.469929 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.473389 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.474937 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.479536 (+ 4599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.480568 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.480701 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.487819 (+ 7118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.487825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.487828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.487861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.492530 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.493903 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.493908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.493912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.494061 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.494509 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.494516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.494523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.494856 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.495088 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.496053 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.496398 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4987,"thread_start_us":4893,"threads_started":1}
I20260927 18:45:25.497225 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.462655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.31.26.195:36097)
0927 18:45:25.469203 (+ 6548us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.469221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.469475 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.470563 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.470583 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.471899 (+ 1316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:25.473117 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.473150 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.476167 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.476176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.479316 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.479338 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.481290 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.481302 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.481778 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.482854 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:25.482885 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:25.487515 (+ 4630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:25.493181 (+ 5666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:25.493186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:25.493189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:25.493747 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:25.494213 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:25.494218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:25.494221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:25.494377 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:25.495574 (+ 1197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:25.495578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:25.495714 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.495866 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.496128 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6384,"thread_start_us":6038,"threads_started":1}
I20260927 18:45:25.497712 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.461127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.31.26.195:49093)
0927 18:45:25.462010 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.462063 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.464823 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.475983 (+ 11160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.475991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.476643 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:25.477041 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.477056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.478864 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.478873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.480892 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.480912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.482305 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.482313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.482911 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.483684 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:25.483728 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:25.487841 (+ 4113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:25.491795 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:25.491807 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:25.491830 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:25.492284 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:25.492679 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:25.492686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:25.492694 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:25.492907 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:25.493611 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:25.493623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:25.494214 (+ 591us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.494559 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.495606 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":380,"threads_started":1}
I20260927 18:45:25.498901 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.461399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:49093 (local address 127.31.26.253:42695)
0927 18:45:25.475516 (+ 14117us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.475525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.475550 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:45:25.475653 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.475661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.475779 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.475915 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.477281 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.478639 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.481180 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.482070 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.484106 (+ 2036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.488108 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.488116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.488121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.488162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.491638 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.492419 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.492425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.492430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.492555 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.493033 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.493038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.493048 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.493450 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.498166 (+ 4716us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.498337 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.498602 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13953,"thread_start_us":13834,"threads_started":1}
I20260927 18:45:25.499538 6786 heartbeater.cc:347] Connected to a master server at 127.31.26.252:39475
I20260927 18:45:25.499914 6786 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:25.500829 6786 heartbeater.cc:511] Master 127.31.26.252:39475 requested a full tablet report, sending...
I20260927 18:45:25.501209 6788 heartbeater.cc:347] Connected to a master server at 127.31.26.253:42695
I20260927 18:45:25.501332 6788 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:25.501834 6788 heartbeater.cc:511] Master 127.31.26.253:42695 requested a full tablet report, sending...
I20260927 18:45:25.502791 6321 ts_manager.cc:194] Registered new tserver with Master: 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441)
I20260927 18:45:25.503237 6134 ts_manager.cc:194] Registered new tserver with Master: 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441)
Sep 27 18:45:25 dist-test-slave-rfd5 krb5kdc[5488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534725, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:25.513242 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.469245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:57311)
0927 18:45:25.485925 (+ 16680us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.485945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.486092 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.486523 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.486529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.488350 (+ 1821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:25.489415 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.489434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.492750 (+ 3316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.492758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.495553 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.495576 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.500059 (+ 4483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.500083 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.500306 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.501052 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:25.501140 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:25.505349 (+ 4209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:25.510235 (+ 4886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:25.510241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:25.510246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:25.510787 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:25.511265 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:25.511271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:25.511274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:25.511444 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:25.512146 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:25.512151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:25.512320 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.512479 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.512778 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16546,"thread_start_us":16369,"threads_started":1}
I20260927 18:45:25.514416 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.469323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:57311 (local address 127.31.26.254:42015)
0927 18:45:25.482951 (+ 13628us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.482958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.486116 (+ 3158us) server_negotiation.cc:407] Connection header received
0927 18:45:25.486225 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.486235 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.486342 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.486469 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.489878 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.491054 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.497690 (+ 6636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.498364 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.498495 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.505675 (+ 7180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.505679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.505683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.505715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.510072 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.510929 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.510938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.510944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.511156 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.511571 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.511578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.511584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.512010 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.512202 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.513204 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.513469 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13497,"spinlock_wait_cycles":6400,"thread_start_us":13651,"threads_started":1}
I20260927 18:45:25.515266 6787 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42015
I20260927 18:45:25.515419 6787 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:25.515883 6787 heartbeater.cc:511] Master 127.31.26.254:42015 requested a full tablet report, sending...
I20260927 18:45:25.516924 5533 ts_manager.cc:194] Re-registered known tserver with Master: 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441)
I20260927 18:45:25.518142 5533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:57311
I20260927 18:45:25.526660 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.505305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.31.26.253:42695)
0927 18:45:25.505473 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.505481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.506051 (+ 570us) server_negotiation.cc:407] Connection header received
0927 18:45:25.506217 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.506222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.506304 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.506417 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.509441 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.511564 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.514846 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.515487 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.515653 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.519309 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.519315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.519322 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.519367 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.523357 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.524298 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.524305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.524309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.524468 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.524952 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.524955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.524957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.525305 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.525564 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.525951 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.526175 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:45:25.528430 6800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.501328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.31.26.254:42015)
0927 18:45:25.504369 (+ 3041us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.504378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.505856 (+ 1478us) server_negotiation.cc:407] Connection header received
0927 18:45:25.506134 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.506140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.506222 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.506297 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.509192 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.510432 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.512372 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.512974 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.513064 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.519139 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.519145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.519149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.519185 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.522896 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.524237 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.524243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.524246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.524392 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.524969 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.524975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.524981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.526620 (+ 1639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.526890 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.527920 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.528113 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2887,"thread_start_us":2771,"threads_started":1}
I20260927 18:45:25.534008 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.507122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.31.26.252:39475)
0927 18:45:25.507312 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.507318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.516701 (+ 9383us) server_negotiation.cc:407] Connection header received
0927 18:45:25.516840 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.516847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.516969 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.517112 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.519389 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.520878 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.522920 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.523384 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.523519 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.527551 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.527557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.527560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.527596 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.530692 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.531543 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.531549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.531552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.531727 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.532432 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.532435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.532438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.532726 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.532986 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.533478 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.533716 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:45:25.532961 5533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:25.541234 5533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:25.592226 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.567547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:41441 (local address 127.0.0.1:40576)
0927 18:45:25.584115 (+ 16568us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.584134 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.584269 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.584820 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.584824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.584846 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:25.585244 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.585262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.586984 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.586992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.589568 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.589582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.591393 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.591402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.591561 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.591587 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.591671 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.591802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16439,"mutex_wait_us":1,"thread_start_us":13737,"threads_started":1}
I20260927 18:45:25.592295 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.568502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.31.26.195:41441)
0927 18:45:25.578070 (+ 9568us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.578085 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.584257 (+ 6172us) server_negotiation.cc:407] Connection header received
0927 18:45:25.584402 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.584408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.584614 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.584789 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:25.585820 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.586852 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.589760 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.591150 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.591293 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.591507 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.591765 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.591945 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9323,"thread_start_us":9103,"threads_started":1}
I20260927 18:45:25.592764 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.565924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.31.26.193:34509)
0927 18:45:25.583930 (+ 18006us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.583940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.583973 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:45:25.584074 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.584082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.584346 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.584539 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:25.585769 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.587368 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.590281 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.591706 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.591852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.592033 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.592283 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.592449 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17811,"thread_start_us":17644,"threads_started":1}
I20260927 18:45:25.592960 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.565619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34509 (local address 127.0.0.1:46730)
0927 18:45:25.566396 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.566424 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.566595 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.585077 (+ 18482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.585114 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.585153 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:25.585597 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.585609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.587513 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.587520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.590117 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.590141 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.591896 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.591902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.592081 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.592112 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.592236 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.592501 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":324,"threads_started":1}
I20260927 18:45:25.593015 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.566458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.31.26.194:43109)
0927 18:45:25.581049 (+ 14591us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.581062 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.582365 (+ 1303us) server_negotiation.cc:407] Connection header received
0927 18:45:25.582525 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.582540 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.582927 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.583223 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:25.584017 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.585580 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.589768 (+ 4188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.590887 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.591034 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.591260 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.592549 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.592742 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14351,"thread_start_us":15590,"threads_started":1}
I20260927 18:45:25.593582 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.566152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:43109 (local address 127.0.0.1:41192)
0927 18:45:25.581664 (+ 15512us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.581694 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.581936 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.583283 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.583292 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.583338 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:25.583817 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.583831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.585771 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.585776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.587791 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.587801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.591108 (+ 3307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.591114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.591292 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.591335 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.591429 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.591576 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15310,"mutex_wait_us":63,"thread_start_us":18132,"threads_started":1}
I20260927 18:45:25.596342 6434 tablet_service.cc:1467] Processing CreateTablet for tablet 947a6319a41e4c649b7491f6943541a0 (DEFAULT_TABLE table=test-table [id=4d9816f3bf54466d83d1808348b8c573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:25.596341 6721 tablet_service.cc:1467] Processing CreateTablet for tablet 947a6319a41e4c649b7491f6943541a0 (DEFAULT_TABLE table=test-table [id=4d9816f3bf54466d83d1808348b8c573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:25.597653 6434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a6319a41e4c649b7491f6943541a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:25.597730 6721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a6319a41e4c649b7491f6943541a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:25.602242 6580 tablet_service.cc:1467] Processing CreateTablet for tablet 947a6319a41e4c649b7491f6943541a0 (DEFAULT_TABLE table=test-table [id=4d9816f3bf54466d83d1808348b8c573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:25.603904 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a6319a41e4c649b7491f6943541a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:25.611778 6822 tablet_bootstrap.cc:492] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Bootstrap starting.
I20260927 18:45:25.613267 6823 tablet_bootstrap.cc:492] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Bootstrap starting.
I20260927 18:45:25.615658 6822 tablet_bootstrap.cc:654] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:25.616703 6823 tablet_bootstrap.cc:654] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:25.617000 6822 log.cc:824] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:25.618067 6823 log.cc:824] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:25.620151 6824 tablet_bootstrap.cc:492] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Bootstrap starting.
I20260927 18:45:25.621560 6822 tablet_bootstrap.cc:492] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: No bootstrap required, opened a new log
I20260927 18:45:25.621866 6823 tablet_bootstrap.cc:492] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: No bootstrap required, opened a new log
I20260927 18:45:25.621856 6822 ts_tablet_manager.cc:1397] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 18:45:25.622115 6823 ts_tablet_manager.cc:1397] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260927 18:45:25.623373 6824 tablet_bootstrap.cc:654] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:25.624490 6824 log.cc:824] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:25.627511 6824 tablet_bootstrap.cc:492] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: No bootstrap required, opened a new log
I20260927 18:45:25.627808 6824 ts_tablet_manager.cc:1397] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 18:45:25.633548 6823 raft_consensus.cc:348] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.633535 6822 raft_consensus.cc:348] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.634068 6822 raft_consensus.cc:374] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:25.634068 6823 raft_consensus.cc:374] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:25.634174 6823 raft_consensus.cc:729] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68fcb20df01479da5a3c24e529aef59, State: Initialized, Role: FOLLOWER
I20260927 18:45:25.634174 6822 raft_consensus.cc:729] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f0bbe3162a4abe8a0b29b10489671c, State: Initialized, Role: FOLLOWER
I20260927 18:45:25.634887 6822 consensus_queue.cc:260] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.634989 6823 consensus_queue.cc:260] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.636317 6824 raft_consensus.cc:348] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.636801 6824 raft_consensus.cc:374] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:25.636902 6824 raft_consensus.cc:729] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70f9613e05e49b79ea32bd28fe43ec8, State: Initialized, Role: FOLLOWER
I20260927 18:45:25.637149 6822 ts_tablet_manager.cc:1428] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 18:45:25.637528 6824 consensus_queue.cc:260] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.641755 6787 heartbeater.cc:503] Master 127.31.26.254:42015 was elected leader, sending a full tablet report...
I20260927 18:45:25.644558 6823 ts_tablet_manager.cc:1428] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Time spent starting tablet: real 0.022s user 0.008s sys 0.006s
I20260927 18:45:25.646772 6831 raft_consensus.cc:482] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:25.647116 6831 raft_consensus.cc:504] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.647641 6824 ts_tablet_manager.cc:1428] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260927 18:45:25.647966 6646 heartbeater.cc:503] Master 127.31.26.254:42015 was elected leader, sending a full tablet report...
I20260927 18:45:25.649258 6831 leader_election.cc:288] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509), 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441)
I20260927 18:45:25.658579 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.649887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53545 (local address 127.31.26.193:34509)
0927 18:45:25.650087 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.650093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.651522 (+ 1429us) server_negotiation.cc:407] Connection header received
0927 18:45:25.651772 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.651777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.651872 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.651963 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:25.652776 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.653875 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.656422 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.657608 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.657786 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.657863 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.658151 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.658289 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:45:25.658676 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.649515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34509 (local address 127.31.26.194:53545)
0927 18:45:25.651416 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.651440 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.651631 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.652149 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.652154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.652182 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:25.652588 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.652603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.654086 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.654092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.656270 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.656283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.657819 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.657825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.657990 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.658018 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.658129 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.658291 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":1013,"threads_started":1}
I20260927 18:45:25.659680 6454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a6319a41e4c649b7491f6943541a0" candidate_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68fcb20df01479da5a3c24e529aef59" is_pre_election: true
I20260927 18:45:25.660323 6454 raft_consensus.cc:2454] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70f9613e05e49b79ea32bd28fe43ec8 in term 0.
I20260927 18:45:25.660774 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.651033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:41441 (local address 127.31.26.194:53553)
0927 18:45:25.654425 (+ 3392us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:25.654444 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:25.654575 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:25.654889 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:25.654894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:25.654917 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:25.655377 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.655388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.656917 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.656921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.658426 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:25.658440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.660017 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.660025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:25.660206 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.660232 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:45:25.660310 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:45:25.660408 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3243,"thread_start_us":3011,"threads_started":1}
I20260927 18:45:25.660768 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.651425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53553 (local address 127.31.26.195:41441)
0927 18:45:25.651687 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.651695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.654536 (+ 2841us) server_negotiation.cc:407] Connection header received
0927 18:45:25.654678 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.654682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.654742 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.654876 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:25.655547 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.656544 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.658574 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.659788 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.659966 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.660047 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.660369 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.660517 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 18:45:25.661206 6533 leader_election.cc:302] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70f9613e05e49b79ea32bd28fe43ec8, d68fcb20df01479da5a3c24e529aef59; no voters:
I20260927 18:45:25.661664 6830 raft_consensus.cc:2781] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:25.661860 6830 raft_consensus.cc:482] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:25.662011 6830 raft_consensus.cc:3037] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:25.662582 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a6319a41e4c649b7491f6943541a0" candidate_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f0bbe3162a4abe8a0b29b10489671c" is_pre_election: true
I20260927 18:45:25.663106 6741 raft_consensus.cc:2454] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70f9613e05e49b79ea32bd28fe43ec8 in term 0.
I20260927 18:45:25.665630 6830 raft_consensus.cc:504] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.666249 6830 leader_election.cc:288] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [CANDIDATE]: Term 1 election: Requested vote from peers d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509), 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441)
I20260927 18:45:25.666757 6454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a6319a41e4c649b7491f6943541a0" candidate_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68fcb20df01479da5a3c24e529aef59"
I20260927 18:45:25.666797 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a6319a41e4c649b7491f6943541a0" candidate_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f0bbe3162a4abe8a0b29b10489671c"
I20260927 18:45:25.666966 6454 raft_consensus.cc:3037] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:25.667006 6741 raft_consensus.cc:3037] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:25.669394 6454 raft_consensus.cc:2454] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70f9613e05e49b79ea32bd28fe43ec8 in term 1.
I20260927 18:45:25.669602 6741 raft_consensus.cc:2454] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70f9613e05e49b79ea32bd28fe43ec8 in term 1.
I20260927 18:45:25.669847 6533 leader_election.cc:302] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70f9613e05e49b79ea32bd28fe43ec8, d68fcb20df01479da5a3c24e529aef59; no voters:
I20260927 18:45:25.670156 6830 raft_consensus.cc:2781] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:25.670353 6830 raft_consensus.cc:686] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 LEADER]: Becoming Leader. State: Replica: b70f9613e05e49b79ea32bd28fe43ec8, State: Running, Role: LEADER
I20260927 18:45:25.670903 6830 consensus_queue.cc:237] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } }
I20260927 18:45:25.676590 5533 catalog_manager.cc:5495] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } health_report { overall_health: UNKNOWN } } }
I20260927 18:45:25.706029 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.700014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.31.26.194:43109)
0927 18:45:25.700253 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.700259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.700398 (+ 139us) server_negotiation.cc:407] Connection header received
0927 18:45:25.700694 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.700699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.700803 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.700968 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:25.701385 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.702063 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.703575 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.703984 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.704181 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.704719 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:25.705410 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:25.705574 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.705645 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.705740 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 18:45:25.710830 6789 tablet.cc:2369] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:25.714905 6454 raft_consensus.cc:1261] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Refusing update from remote peer b70f9613e05e49b79ea32bd28fe43ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:25.714905 6741 raft_consensus.cc:1261] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Refusing update from remote peer b70f9613e05e49b79ea32bd28fe43ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:25.716006 6830 consensus_queue.cc:1035] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:25.716329 6831 consensus_queue.cc:1035] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:45:25.726217 6502 tablet.cc:2369] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:25.733625 6836 mvcc.cc:205] Tried to move back new op lower bound from 7334030236517761024 to 7334030236352749568. Current Snapshot: MvccSnapshot[applied={T|T < 7334030236517761024}]
I20260927 18:45:25.738059 6838 mvcc.cc:205] Tried to move back new op lower bound from 7334030236517761024 to 7334030236352749568. Current Snapshot: MvccSnapshot[applied={T|T < 7334030236517761024}]
I20260927 18:45:25.763689 5533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:25.763974 5533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:25.771245 5533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bd5ba21bf314b60a28df0efa8261ff7: Sending DeleteTablet for 3 replicas of tablet 947a6319a41e4c649b7491f6943541a0
I20260927 18:45:25.772485 6434 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a6319a41e4c649b7491f6943541a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:25 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:46730
I20260927 18:45:25.772631 6721 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a6319a41e4c649b7491f6943541a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:25 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:40576
I20260927 18:45:25.772869 6580 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a6319a41e4c649b7491f6943541a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:25 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:41192
I20260927 18:45:25.780540 6846 tablet_replica.cc:330] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: stopping tablet replica
I20260927 18:45:25.781018 6846 raft_consensus.cc:2229] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:25.782052 6846 raft_consensus.cc:2258] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:25.784209 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5501
I20260927 18:45:25.785552 5563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:25.786238 6845 tablet_replica.cc:330] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: stopping tablet replica
I20260927 18:45:25.786839 6845 raft_consensus.cc:2229] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:25.787469 6845 raft_consensus.cc:2258] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:25.788385 6847 tablet_replica.cc:330] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: stopping tablet replica
I20260927 18:45:25.788975 6847 raft_consensus.cc:2229] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:25.789737 6845 ts_tablet_manager.cc:1905] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:25.789822 6847 raft_consensus.cc:2258] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:25.791992 6846 ts_tablet_manager.cc:1905] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:25.795949 6847 ts_tablet_manager.cc:1905] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:25.796491 6845 ts_tablet_manager.cc:1918] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:25.796693 6845 log.cc:1196] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/947a6319a41e4c649b7491f6943541a0
I20260927 18:45:25.797314 6845 ts_tablet_manager.cc:1939] T 947a6319a41e4c649b7491f6943541a0 P d68fcb20df01479da5a3c24e529aef59: Deleting consensus metadata
I20260927 18:45:25.802325 6847 ts_tablet_manager.cc:1918] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:25.802649 6847 log.cc:1196] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/947a6319a41e4c649b7491f6943541a0
I20260927 18:45:25.803349 6847 ts_tablet_manager.cc:1939] T 947a6319a41e4c649b7491f6943541a0 P b70f9613e05e49b79ea32bd28fe43ec8: Deleting consensus metadata
I20260927 18:45:25.807313 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.779728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.31.26.253:42695)
0927 18:45:25.779962 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.779971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.782446 (+ 2475us) server_negotiation.cc:407] Connection header received
0927 18:45:25.782799 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.782806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.782892 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.782963 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.790345 (+ 7382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.791529 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.793959 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.794623 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.794733 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.798886 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.798894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.798899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.798946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.803012 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.804286 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.804296 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.804302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.804499 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.805176 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.805185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.805191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.805677 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.805968 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.806657 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.806908 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:45:25.811581 6846 ts_tablet_manager.cc:1918] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:25.811805 6846 log.cc:1196] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/947a6319a41e4c649b7491f6943541a0
I20260927 18:45:25.812413 6846 ts_tablet_manager.cc:1939] T 947a6319a41e4c649b7491f6943541a0 P 35f0bbe3162a4abe8a0b29b10489671c: Deleting consensus metadata
I20260927 18:45:25.825493 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.783204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.31.26.252:39475)
0927 18:45:25.783421 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.783430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.808764 (+ 25334us) server_negotiation.cc:407] Connection header received
0927 18:45:25.809032 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.809037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.809146 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.809218 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.810900 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.811645 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.813149 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.813814 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.813949 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.817950 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.817960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.817967 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.818017 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.822007 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.822986 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.822994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.822999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.823171 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.823682 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.823686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.823689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.824094 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.824389 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.824765 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.825061 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:45:25.929441 5517 catalog_manager.cc:4841] TS d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509): tablet 947a6319a41e4c649b7491f6943541a0 (table test-table [id=4d9816f3bf54466d83d1808348b8c573]) successfully deleted
W20260927 18:45:25.929795 5517 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.26.193:34509, user_credentials={real_user=slave}} blocked reactor thread for 130532us
I20260927 18:45:25.930335 5517 catalog_manager.cc:4841] TS 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441): tablet 947a6319a41e4c649b7491f6943541a0 (table test-table [id=4d9816f3bf54466d83d1808348b8c573]) successfully deleted
I20260927 18:45:25.930528 5518 catalog_manager.cc:4841] TS b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109): tablet 947a6319a41e4c649b7491f6943541a0 (table test-table [id=4d9816f3bf54466d83d1808348b8c573]) successfully deleted
W20260927 18:45:25.930670 5518 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.26.194:43109, user_credentials={real_user=slave}} blocked reactor thread for 125341us
I20260927 18:45:25.930895 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5501
I20260927 18:45:25.978667 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.960455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.31.26.253:42695)
0927 18:45:25.960673 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.960680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.960702 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:45:25.960767 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.960773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.960856 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.960965 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.963464 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.964522 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.966648 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.967109 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.967189 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.972130 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.972136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.972139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.972178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.975416 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.976269 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.976275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.976280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.976440 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.976869 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.976879 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.976881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.977372 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.977611 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.978057 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.978305 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:45:25.979055 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:25.960348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.31.26.252:39475)
0927 18:45:25.960589 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:25.960596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:25.960830 (+ 234us) server_negotiation.cc:407] Connection header received
0927 18:45:25.961012 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:25.961019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:25.961143 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:25.961259 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:25.963224 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.964357 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.966639 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:25.967348 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:25.967481 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:25.972098 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:25.972106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:25.972111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:25.972163 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:25.975955 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.976680 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.976686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.976690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.976843 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:25.977292 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:25.977297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:25.977301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:25.977675 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:25.977847 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:25.978468 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 18:45:25.978706 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:45:26.749348 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:26.748494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:39799)
0927 18:45:26.748970 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:26.749141 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
W20260927 18:45:26.749838 6500 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:42015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:26.759649 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:26.758822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:57011)
0927 18:45:26.759280 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:26.759427 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":230,"threads_started":1}
W20260927 18:45:26.760071 6787 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:42015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:26.760854 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:26.760179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.194:55917)
0927 18:45:26.760560 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:26.760656 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":199,"threads_started":1}
W20260927 18:45:26.761260 6646 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:42015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:27.270866 6859 raft_consensus.cc:482] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2bd5ba21bf314b60a28df0efa8261ff7)
I20260927 18:45:27.271202 6859 raft_consensus.cc:504] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } }
I20260927 18:45:27.271752 6860 raft_consensus.cc:482] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2bd5ba21bf314b60a28df0efa8261ff7)
I20260927 18:45:27.272001 6860 raft_consensus.cc:504] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } }
I20260927 18:45:27.272868 6859 leader_election.cc:288] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015), a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252:39475)
I20260927 18:45:27.273485 6860 leader_election.cc:288] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015), debbbd3ea27044f6b6ef62e85d86cde6 (127.31.26.253:42695)
I20260927 18:45:27.274224 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.272953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34958)
0927 18:45:27.273549 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.273905 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
W20260927 18:45:27.274571 6120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:27.275600 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.273591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34968)
0927 18:45:27.275043 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.275255 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":1183,"threads_started":1}
W20260927 18:45:27.276518 6307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:27.286214 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.273652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:39475 (local address 127.0.0.1:39714)
0927 18:45:27.274279 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.274304 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.274472 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.276462 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.276466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.276504 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.276932 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.276952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.279371 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.279381 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.282452 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.282479 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.284866 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.284876 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.285197 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.285266 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.285375 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.285617 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":636,"threads_started":1}
I20260927 18:45:27.287297 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.286426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34976)
0927 18:45:27.286911 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.287048 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":38}
I20260927 18:45:27.287614 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.274163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.31.26.252:39475)
0927 18:45:27.274936 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.274945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.274964 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:45:27.275044 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.275053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.275158 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.277195 (+ 2037us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.277508 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.279154 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.282773 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.284599 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.286609 (+ 2010us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.286778 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.286854 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.287061 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":759,"threads_started":1}
W20260927 18:45:27.287554 6120 leader_election.cc:334] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015): Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:27.288405 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" is_pre_election: true
I20260927 18:45:27.288863 6331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate debbbd3ea27044f6b6ef62e85d86cde6 in term 1.
I20260927 18:45:27.289460 6120 leader_election.cc:302] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a165b7bf2db84ddcbd3cb2b6482c55b1, debbbd3ea27044f6b6ef62e85d86cde6; no voters: 2bd5ba21bf314b60a28df0efa8261ff7
I20260927 18:45:27.289880 6859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:45:27.289937 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.289358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34978)
0927 18:45:27.289578 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.289741 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 18:45:27.290006 6859 raft_consensus.cc:482] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2bd5ba21bf314b60a28df0efa8261ff7)
I20260927 18:45:27.290092 6859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 1 FOLLOWER]: Advancing to term 2
W20260927 18:45:27.290128 6307 leader_election.cc:334] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015): Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:27.292440 6859 raft_consensus.cc:504] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } }
I20260927 18:45:27.293303 6859 leader_election.cc:288] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [CANDIDATE]: Term 2 election: Requested vote from peers 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015), a165b7bf2db84ddcbd3cb2b6482c55b1 (127.31.26.252:39475)
I20260927 18:45:27.293920 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.293358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34982)
0927 18:45:27.293592 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.293701 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 18:45:27.294021 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1"
I20260927 18:45:27.294265 6331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:45:27.294598 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.275131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:42695 (local address 127.0.0.1:53154)
0927 18:45:27.275840 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.275862 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.276003 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.288582 (+ 12579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.288587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.288624 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.288913 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.288922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.290599 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.290603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.291945 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.291958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.293683 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.293690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.293906 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.293944 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.294065 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.294242 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":1116,"threads_started":1}
I20260927 18:45:27.294731 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.275465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.31.26.253:42695)
0927 18:45:27.288279 (+ 12814us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.288284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.288311 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:45:27.288376 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.288381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.288460 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.288569 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.289139 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.290440 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.292127 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.293466 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.293610 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.293735 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.294117 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.294260 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12690,"thread_start_us":12476,"threads_started":1}
I20260927 18:45:27.295367 6144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" is_pre_election: true
I20260927 18:45:27.295571 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.295162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:34984)
0927 18:45:27.295309 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.295414 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:45:27.295758 6144 raft_consensus.cc:2379] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a165b7bf2db84ddcbd3cb2b6482c55b1 in current term 2: Already voted for candidate debbbd3ea27044f6b6ef62e85d86cde6 in this term.
W20260927 18:45:27.295781 6120 leader_election.cc:334] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015): Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
I20260927 18:45:27.296401 6307 leader_election.cc:302] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a165b7bf2db84ddcbd3cb2b6482c55b1; no voters: 2bd5ba21bf314b60a28df0efa8261ff7, debbbd3ea27044f6b6ef62e85d86cde6
I20260927 18:45:27.297348 6331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate debbbd3ea27044f6b6ef62e85d86cde6 in term 2.
I20260927 18:45:27.297569 6860 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:45:27.297791 6120 leader_election.cc:302] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a165b7bf2db84ddcbd3cb2b6482c55b1, debbbd3ea27044f6b6ef62e85d86cde6; no voters: 2bd5ba21bf314b60a28df0efa8261ff7
I20260927 18:45:27.298066 6859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:45:27.298547 6859 raft_consensus.cc:686] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [term 2 LEADER]: Becoming Leader. State: Replica: debbbd3ea27044f6b6ef62e85d86cde6, State: Running, Role: LEADER
I20260927 18:45:27.298959 6859 consensus_queue.cc:237] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } }
I20260927 18:45:27.300763 6869 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debbbd3ea27044f6b6ef62e85d86cde6. Latest consensus state: current_term: 2 leader_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:27.301146 6869 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:27.301780 6871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:27.305701 6871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:27.306582 6871 catalog_manager.cc:1259] Loaded cluster ID: 6824644ba5254436b842b4f7238e3b8b
I20260927 18:45:27.306694 6871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:27.307873 6871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:27.308686 6871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Loaded TSK: 0
I20260927 18:45:27.309625 6871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:45:27.334070 6133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:27.341717 6133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:27.346108 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.345671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:35000)
0927 18:45:27.345873 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.345960 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:45:27.346541 6331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [term 2 FOLLOWER]: Refusing update from remote peer debbbd3ea27044f6b6ef62e85d86cde6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 18:45:27.347340 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.346902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:35016)
0927 18:45:27.347050 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.347131 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:45:27.347450 6869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 18:45:27.347805 6120 consensus_peers.cc:487] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 -> Peer 2bd5ba21bf314b60a28df0efa8261ff7 (127.31.26.254:42015): Couldn't send request to peer 2bd5ba21bf314b60a28df0efa8261ff7. Status: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:45:27.350560 6860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debbbd3ea27044f6b6ef62e85d86cde6. Latest consensus state: current_term: 2 leader_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:27.350908 6860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:45:27.352615 6869 sys_catalog.cc:455] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:27.352977 6869 sys_catalog.cc:458] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:27.353948 6860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2bd5ba21bf314b60a28df0efa8261ff7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42015 } } peers { permanent_uuid: "debbbd3ea27044f6b6ef62e85d86cde6" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 42695 } attrs { promote: false } } peers { permanent_uuid: "a165b7bf2db84ddcbd3cb2b6482c55b1" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 39475 } attrs { promote: false } } }
I20260927 18:45:27.354205 6860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a165b7bf2db84ddcbd3cb2b6482c55b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:45:27.367858 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.361937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:43109 (local address 127.0.0.1:41210)
0927 18:45:27.362081 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.362102 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.362249 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.363108 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.363113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.363140 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.363414 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.363422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.364385 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.364392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.365955 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.365966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.367101 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.367106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.367274 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.367297 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.367417 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.367581 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:45:27.368088 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.362316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.31.26.194:43109)
0927 18:45:27.362786 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.362792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.362811 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:45:27.362894 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.362901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.362984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.363090 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.363551 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.364255 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.366110 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.366888 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.367039 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.367160 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.367520 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.367743 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":517,"threads_started":1}
I20260927 18:45:27.369201 6580 tablet_service.cc:1467] Processing CreateTablet for tablet c5bd859650b54d56947ecc1091d24a88 (DEFAULT_TABLE table=test-table [id=4093ac0615dc4b5aaa5494dc3126b21b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:27.369705 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5bd859650b54d56947ecc1091d24a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:27.374115 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.361341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:41441 (local address 127.0.0.1:40584)
0927 18:45:27.362281 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.362298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.362371 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.368508 (+ 6137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.368513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.368542 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.368755 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.368767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.370204 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.370209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.371777 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.371791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.373353 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.373360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.373536 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.373574 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.373640 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.373800 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":27}
I20260927 18:45:27.374972 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.361592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.31.26.195:41441)
0927 18:45:27.368107 (+ 6515us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.368115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.368138 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:45:27.368209 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.368217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.368314 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.368443 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.368919 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.370056 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.371952 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.373163 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.374365 (+ 1202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.374482 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.374535 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.374674 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6388,"thread_start_us":267,"threads_started":1}
I20260927 18:45:27.374993 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.362200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34509 (local address 127.0.0.1:46744)
0927 18:45:27.368130 (+ 5930us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.368148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.368260 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.368927 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.368931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.368958 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.369229 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.369241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.371274 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.371280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.372840 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.372854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.374380 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.374387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.374582 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.374606 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.374656 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.374756 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5828,"mutex_wait_us":21,"thread_start_us":7077,"threads_started":1}
I20260927 18:45:27.375759 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.362654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.31.26.193:34509)
0927 18:45:27.365471 (+ 2817us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.365477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.368377 (+ 2900us) server_negotiation.cc:407] Connection header received
0927 18:45:27.368463 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.368471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.368554 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.368629 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.369678 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.371097 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.373034 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.374087 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.375237 (+ 1150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.375341 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.375395 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.375524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2702,"thread_start_us":2575,"threads_started":1}
I20260927 18:45:27.375744 6721 tablet_service.cc:1467] Processing CreateTablet for tablet c5bd859650b54d56947ecc1091d24a88 (DEFAULT_TABLE table=test-table [id=4093ac0615dc4b5aaa5494dc3126b21b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:27.376295 6721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5bd859650b54d56947ecc1091d24a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:27.376837 6882 tablet_bootstrap.cc:492] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Bootstrap starting.
I20260927 18:45:27.377981 6434 tablet_service.cc:1467] Processing CreateTablet for tablet c5bd859650b54d56947ecc1091d24a88 (DEFAULT_TABLE table=test-table [id=4093ac0615dc4b5aaa5494dc3126b21b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:27.378418 6434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5bd859650b54d56947ecc1091d24a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:27.378549 6882 tablet_bootstrap.cc:654] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:27.380559 6882 tablet_bootstrap.cc:492] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: No bootstrap required, opened a new log
I20260927 18:45:27.380730 6882 ts_tablet_manager.cc:1397] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:45:27.381534 6882 raft_consensus.cc:348] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.381737 6882 raft_consensus.cc:374] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:27.381825 6882 raft_consensus.cc:729] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70f9613e05e49b79ea32bd28fe43ec8, State: Initialized, Role: FOLLOWER
I20260927 18:45:27.382095 6882 consensus_queue.cc:260] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.382932 6882 ts_tablet_manager.cc:1428] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:45:27.384536 6885 tablet_bootstrap.cc:492] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Bootstrap starting.
I20260927 18:45:27.386125 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.383623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.194:56121)
0927 18:45:27.385597 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.385776 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1775,"thread_start_us":1626,"threads_started":1}
I20260927 18:45:27.386585 6885 tablet_bootstrap.cc:654] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:27.386772 6887 tablet_bootstrap.cc:492] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Bootstrap starting.
I20260927 18:45:27.388216 6887 tablet_bootstrap.cc:654] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:27.394891 6887 tablet_bootstrap.cc:492] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: No bootstrap required, opened a new log
I20260927 18:45:27.395030 6885 tablet_bootstrap.cc:492] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: No bootstrap required, opened a new log
I20260927 18:45:27.395072 6887 ts_tablet_manager.cc:1397] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 18:45:27.395193 6885 ts_tablet_manager.cc:1397] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260927 18:45:27.395816 6887 raft_consensus.cc:348] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.395926 6885 raft_consensus.cc:348] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.396060 6887 raft_consensus.cc:374] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:27.396164 6885 raft_consensus.cc:374] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:27.396171 6887 raft_consensus.cc:729] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68fcb20df01479da5a3c24e529aef59, State: Initialized, Role: FOLLOWER
I20260927 18:45:27.396286 6885 raft_consensus.cc:729] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f0bbe3162a4abe8a0b29b10489671c, State: Initialized, Role: FOLLOWER
I20260927 18:45:27.396442 6887 consensus_queue.cc:260] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.396528 6885 consensus_queue.cc:260] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.397342 6885 ts_tablet_manager.cc:1428] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 18:45:27.397785 6887 ts_tablet_manager.cc:1428] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 18:45:27.399057 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.397982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:40049)
0927 18:45:27.398550 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.398766 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260927 18:45:27.404752 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.399688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:44501)
0927 18:45:27.404513 (+ 4825us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.404604 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4737,"threads_started":1}
I20260927 18:45:27.406070 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.398344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:59107)
0927 18:45:27.405671 (+ 7327us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.405811 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7177,"thread_start_us":7007,"threads_started":1}
I20260927 18:45:27.407241 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.406819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:55941)
0927 18:45:27.407013 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.407093 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 18:45:27.407601 6788 heartbeater.cc:503] Master 127.31.26.253:42695 was elected leader, sending a full tablet report...
W20260927 18:45:27.606402 6648 tablet.cc:2369] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:27.667749 6891 raft_consensus.cc:482] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:27.668056 6891 raft_consensus.cc:504] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.669720 6891 leader_election.cc:288] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109), d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509)
I20260927 18:45:27.677307 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.670145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:50377 (local address 127.31.26.193:34509)
0927 18:45:27.670455 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.670464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.670930 (+ 466us) server_negotiation.cc:407] Connection header received
0927 18:45:27.671082 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.671085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.671139 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.671209 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.671777 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.673027 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.675493 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.676314 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.676504 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.676590 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.676867 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.676992 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 18:45:27.677350 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.670126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:58249 (local address 127.31.26.194:43109)
0927 18:45:27.670365 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.670370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.670391 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:45:27.670632 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.670639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.670742 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.670836 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:27.671366 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.672912 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.675527 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.676412 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.676617 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.676693 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.676940 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.677066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:45:27.677605 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.669992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:43109 (local address 127.31.26.195:58249)
0927 18:45:27.670222 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.670243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.670398 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.670889 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.670896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.670945 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.671219 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.671232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.673207 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.673216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.675344 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.675358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.676607 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.676613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.676788 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.676810 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.676915 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.677066 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":145,"spinlock_wait_cycles":384}
I20260927 18:45:27.677873 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.670006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34509 (local address 127.31.26.195:50377)
0927 18:45:27.670799 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.670840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:27.670959 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:27.671254 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:27.671261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:27.671299 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:27.671533 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.671540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.673187 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.673196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.675321 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:27.675338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.676504 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.676511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:27.676702 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.676734 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:45:27.676840 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:45:27.676985 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":346,"spinlock_wait_cycles":11520,"thread_start_us":379,"threads_started":1}
I20260927 18:45:27.678465 6454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bd859650b54d56947ecc1091d24a88" candidate_uuid: "35f0bbe3162a4abe8a0b29b10489671c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68fcb20df01479da5a3c24e529aef59" is_pre_election: true
I20260927 18:45:27.678472 6600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bd859650b54d56947ecc1091d24a88" candidate_uuid: "35f0bbe3162a4abe8a0b29b10489671c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" is_pre_election: true
I20260927 18:45:27.678738 6454 raft_consensus.cc:2454] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f0bbe3162a4abe8a0b29b10489671c in term 0.
I20260927 18:45:27.678834 6600 raft_consensus.cc:2454] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f0bbe3162a4abe8a0b29b10489671c in term 0.
I20260927 18:45:27.679322 6674 leader_election.cc:302] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f0bbe3162a4abe8a0b29b10489671c, d68fcb20df01479da5a3c24e529aef59; no voters:
I20260927 18:45:27.679744 6891 raft_consensus.cc:2781] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:27.679878 6891 raft_consensus.cc:482] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:27.679980 6891 raft_consensus.cc:3037] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:27.682164 6891 raft_consensus.cc:504] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.682808 6891 leader_election.cc:288] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [CANDIDATE]: Term 1 election: Requested vote from peers b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109), d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509)
I20260927 18:45:27.683182 6600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bd859650b54d56947ecc1091d24a88" candidate_uuid: "35f0bbe3162a4abe8a0b29b10489671c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70f9613e05e49b79ea32bd28fe43ec8"
I20260927 18:45:27.683391 6600 raft_consensus.cc:3037] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:27.683327 6454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bd859650b54d56947ecc1091d24a88" candidate_uuid: "35f0bbe3162a4abe8a0b29b10489671c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68fcb20df01479da5a3c24e529aef59"
I20260927 18:45:27.683557 6454 raft_consensus.cc:3037] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:27.685643 6600 raft_consensus.cc:2454] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f0bbe3162a4abe8a0b29b10489671c in term 1.
I20260927 18:45:27.685662 6454 raft_consensus.cc:2454] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f0bbe3162a4abe8a0b29b10489671c in term 1.
I20260927 18:45:27.686136 6675 leader_election.cc:302] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f0bbe3162a4abe8a0b29b10489671c, b70f9613e05e49b79ea32bd28fe43ec8; no voters:
I20260927 18:45:27.686476 6891 raft_consensus.cc:2781] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:27.686928 6891 raft_consensus.cc:686] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 LEADER]: Becoming Leader. State: Replica: 35f0bbe3162a4abe8a0b29b10489671c, State: Running, Role: LEADER
I20260927 18:45:27.687388 6891 consensus_queue.cc:237] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } }
I20260927 18:45:27.688005 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.687546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:58825)
0927 18:45:27.687753 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.687867 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:45:27.690968 6134 catalog_manager.cc:5495] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195). New cstate: current_term: 1 leader_uuid: "35f0bbe3162a4abe8a0b29b10489671c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f0bbe3162a4abe8a0b29b10489671c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 } health_report { overall_health: UNKNOWN } } }
I20260927 18:45:27.757040 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.751681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.31.26.195:41441)
0927 18:45:27.751911 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.751916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.751933 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:45:27.752021 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.752025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.752106 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.752238 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:27.752539 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.753385 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.754863 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.755391 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.755491 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.755898 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:27.756490 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:27.756598 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.756682 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.756750 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:45:27.761521 6454 raft_consensus.cc:1261] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Refusing update from remote peer 35f0bbe3162a4abe8a0b29b10489671c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:27.761874 6600 raft_consensus.cc:1261] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Refusing update from remote peer 35f0bbe3162a4abe8a0b29b10489671c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:27.762296 6896 consensus_queue.cc:1035] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68fcb20df01479da5a3c24e529aef59" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:27.762712 6891 consensus_queue.cc:1035] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70f9613e05e49b79ea32bd28fe43ec8" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:27.764367 6501 heartbeater.cc:503] Master 127.31.26.253:42695 was elected leader, sending a full tablet report...
I20260927 18:45:27.765674 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.764893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:37791)
0927 18:45:27.765211 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.765401 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 18:45:27.765650 6647 heartbeater.cc:503] Master 127.31.26.253:42695 was elected leader, sending a full tablet report...
I20260927 18:45:27.766515 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.766063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.194:54311)
0927 18:45:27.766295 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.766370 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 18:45:27.772251 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.771653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.194:51247)
0927 18:45:27.771856 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.772008 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 18:45:27.775264 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.774748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:50183)
0927 18:45:27.774965 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.775056 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 18:45:27.784725 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.784241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.193:35749)
0927 18:45:27.784471 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.784562 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 18:45:27.784809 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.784245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.31.26.195:45569)
0927 18:45:27.784529 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.784652 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260927 18:45:27.787590 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.786999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:35018)
0927 18:45:27.787322 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.787435 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 18:45:27.788595 6134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:27.788867 6134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:27.789562 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.788312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:35026)
0927 18:45:27.789217 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:27.789325 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":804}
I20260927 18:45:27.794484 6134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P debbbd3ea27044f6b6ef62e85d86cde6: Sending DeleteTablet for 3 replicas of tablet c5bd859650b54d56947ecc1091d24a88
I20260927 18:45:27.795742 6580 tablet_service.cc:1514] Processing DeleteTablet for tablet c5bd859650b54d56947ecc1091d24a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:27 UTC) from {username='oryx', principal='oryx/127.31.26.253@KRBTEST.COM'} at 127.0.0.1:41210
I20260927 18:45:27.796038 6721 tablet_service.cc:1514] Processing DeleteTablet for tablet c5bd859650b54d56947ecc1091d24a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:27 UTC) from {username='oryx', principal='oryx/127.31.26.253@KRBTEST.COM'} at 127.0.0.1:40584
I20260927 18:45:27.796469 6434 tablet_service.cc:1514] Processing DeleteTablet for tablet c5bd859650b54d56947ecc1091d24a88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:27 UTC) from {username='oryx', principal='oryx/127.31.26.253@KRBTEST.COM'} at 127.0.0.1:46744
I20260927 18:45:27.797358 6910 tablet_replica.cc:330] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: stopping tablet replica
I20260927 18:45:27.797598 6910 raft_consensus.cc:2229] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:27.797865 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 6301
I20260927 18:45:27.797927 6910 raft_consensus.cc:2258] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:27.799117 6910 ts_tablet_manager.cc:1905] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:27.806123 6910 ts_tablet_manager.cc:1918] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:27.806391 6910 log.cc:1196] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/c5bd859650b54d56947ecc1091d24a88
I20260927 18:45:27.806972 6910 ts_tablet_manager.cc:1939] T c5bd859650b54d56947ecc1091d24a88 P d68fcb20df01479da5a3c24e529aef59: Deleting consensus metadata
I20260927 18:45:27.808398 6118 catalog_manager.cc:4841] TS d68fcb20df01479da5a3c24e529aef59 (127.31.26.193:34509): tablet c5bd859650b54d56947ecc1091d24a88 (table test-table [id=4093ac0615dc4b5aaa5494dc3126b21b]) successfully deleted
I20260927 18:45:27.810333 6909 tablet_replica.cc:330] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: stopping tablet replica
I20260927 18:45:27.812534 6909 raft_consensus.cc:2229] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:27.813369 6909 raft_consensus.cc:2258] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:27.814098 6908 tablet_replica.cc:330] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: stopping tablet replica
I20260927 18:45:27.814249 6908 raft_consensus.cc:2229] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:27.814474 6908 raft_consensus.cc:2258] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:27.815474 6908 ts_tablet_manager.cc:1905] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:27.816634 6909 ts_tablet_manager.cc:1905] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:27.822388 6908 ts_tablet_manager.cc:1918] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:27.822542 6908 log.cc:1196] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/c5bd859650b54d56947ecc1091d24a88
I20260927 18:45:27.823093 6908 ts_tablet_manager.cc:1939] T c5bd859650b54d56947ecc1091d24a88 P b70f9613e05e49b79ea32bd28fe43ec8: Deleting consensus metadata
I20260927 18:45:27.824113 6909 ts_tablet_manager.cc:1918] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:27.824335 6909 log.cc:1196] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/c5bd859650b54d56947ecc1091d24a88
I20260927 18:45:27.824358 6119 catalog_manager.cc:4841] TS b70f9613e05e49b79ea32bd28fe43ec8 (127.31.26.194:43109): tablet c5bd859650b54d56947ecc1091d24a88 (table test-table [id=4093ac0615dc4b5aaa5494dc3126b21b]) successfully deleted
I20260927 18:45:27.824991 6909 ts_tablet_manager.cc:1939] T c5bd859650b54d56947ecc1091d24a88 P 35f0bbe3162a4abe8a0b29b10489671c: Deleting consensus metadata
I20260927 18:45:27.826251 6118 catalog_manager.cc:4841] TS 35f0bbe3162a4abe8a0b29b10489671c (127.31.26.195:41441): tablet c5bd859650b54d56947ecc1091d24a88 (table test-table [id=4093ac0615dc4b5aaa5494dc3126b21b]) successfully deleted
I20260927 18:45:27.828286 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.800430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.31.26.193:34509)
0927 18:45:27.800672 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.800680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.809705 (+ 9025us) server_negotiation.cc:407] Connection header received
0927 18:45:27.809903 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.809911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.810024 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.810158 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:27.812395 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.813772 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.816072 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.816694 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.816834 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:27.820662 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:27.820682 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:27.820706 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:27.820754 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:27.824892 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:27.825758 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:27.825769 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:27.825777 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:27.825907 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:27.826582 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:27.826589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:27.826593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:27.827066 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:27.827399 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:27.827654 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:45:27.827906 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:45:27.828977 6494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:27.971984 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 6301
I20260927 18:45:27.989244 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 6514
I20260927 18:45:28.013295 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:27.990072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.31.26.194:43109)
0927 18:45:27.990267 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:27.990273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:27.990296 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:45:27.990477 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:27.990483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:27.990563 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:27.990674 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:27.991970 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.993061 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.994488 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:27.995123 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:27.995264 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:28.005780 (+ 10516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:28.005803 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:28.005826 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:28.005885 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:28.009937 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:28.010919 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:28.010930 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:28.010934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:28.011103 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:28.011554 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:28.011561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:28.011566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:28.012035 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:28.012352 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:28.012642 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:45:28.012868 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:45:28.014261 6640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:28.142570 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 6514
I20260927 18:45:28.169872 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 6658
I20260927 18:45:28.184895 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:28.170726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.31.26.195:41441)
0927 18:45:28.170988 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:28.170994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:28.171017 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:45:28.171093 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:28.171099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:28.171238 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:28.171359 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:28.172949 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:28.173918 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:28.175131 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:28.175561 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:28.175706 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:28.178645 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:28.178668 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:28.178685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:28.178719 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:28.181897 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:28.182645 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:28.182656 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:28.182662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:28.182848 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:28.183254 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:28.183261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:28.183269 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:28.183678 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:28.183996 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:28.184307 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 18:45:28.184527 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 18:45:28.185716 6781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:28.317978 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 6658
I20260927 18:45:28.338953 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 5995
I20260927 18:45:28.339011 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:28.337193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:35036)
0927 18:45:28.338725 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:28.338836 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1373,"thread_start_us":1279,"threads_started":1}
I20260927 18:45:28.340065 6164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:28.340389 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:28.339999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42015 (local address 127.0.0.1:35046)
0927 18:45:28.340168 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:28.340267 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:45:28.459817 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 5995
W20260927 18:45:28.480636 6306 connection.cc:535] server connection from 127.0.0.1:39714 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:45:28.481014 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 6095
I20260927 18:45:28.482090 6351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:28.596385 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 6095
2026-09-27T18:45:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9938 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:28 dist-test-slave-rfd5 krb5kdc[6922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:28 dist-test-slave-rfd5 krb5kdc[6922](info): set up 2 sockets
Sep 27 18:45:28 dist-test-slave-rfd5 krb5kdc[6922](info): commencing operation
krb5kdc: starting...
W20260927 18:45:29.710238 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:29 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab.
Entry for principal oryx/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab.
I20260927 18:45:29.937791 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42181
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab
--principal=oryx/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:30.279937 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:30.280273 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:30.280439 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:30.287984 6935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:30.288123 6935 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:45:30.288213 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:30.288328 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:30.288430 6935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:30.288544 6935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:30.298895 6935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:42181
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.254.keytab
--principal=oryx/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:30.300287 6935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:30.301780 6935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:30.314177 6943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:30.314240 6941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:30.314873 6940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:30.315249 6935 server_base.cc:1034] running on GCE node
I20260927 18:45:30.316040 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:30.317653 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:30.318883 6935 hybrid_clock.cc:648] HybridClock initialized: now 1790534730318818 us; error 36 us; skew 500 ppm
I20260927 18:45:30.319258 6935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:30 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:30.322512 6935 init.cc:377] Logged in from keytab as oryx/127.31.26.254@KRBTEST.COM (short username oryx)
I20260927 18:45:30.324041 6935 webserver.cc:460] Webserver started at http://127.31.26.254:46087/ using document root <none> and password file <none>
I20260927 18:45:30.324782 6935 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:30.324915 6935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:30.325309 6935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:30.328346 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "85e00576785742d98da3518458c87156"
format_stamp: "Formatted at 2026-09-27 18:45:30 on dist-test-slave-rfd5"
server_key: "99b545550a1cc8ad49af5842472e5d1b"
server_key_iv: "72401e9587a0647b7ce0cfcbe07792ec"
server_key_version: "encryptionkey@0"
I20260927 18:45:30.329321 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "85e00576785742d98da3518458c87156"
format_stamp: "Formatted at 2026-09-27 18:45:30 on dist-test-slave-rfd5"
server_key: "99b545550a1cc8ad49af5842472e5d1b"
server_key_iv: "72401e9587a0647b7ce0cfcbe07792ec"
server_key_version: "encryptionkey@0"
I20260927 18:45:30.334601 6935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 18:45:30.338371 6950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:30.339841 6935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:30.340042 6935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "85e00576785742d98da3518458c87156"
format_stamp: "Formatted at 2026-09-27 18:45:30 on dist-test-slave-rfd5"
server_key: "99b545550a1cc8ad49af5842472e5d1b"
server_key_iv: "72401e9587a0647b7ce0cfcbe07792ec"
server_key_version: "encryptionkey@0"
I20260927 18:45:30.340332 6935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:30.359536 6935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:30.364166 6935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:30.364473 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:30.387117 6935 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:42181
I20260927 18:45:30.387137 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:42181 every 8 connection(s)
I20260927 18:45:30.388496 6935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:30.390420 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 6935
I20260927 18:45:30.390637 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:30.391294 31851 external_mini_cluster.cc:1428] Setting key b39f6f7f2036e287638572686d047731
I20260927 18:45:30.392679 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:45:30 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:30.407495 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Bootstrap starting.
I20260927 18:45:30.411042 7012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:30.412228 7012 log.cc:824] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:30.415534 7012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: No bootstrap required, opened a new log
I20260927 18:45:30.415627 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:30.393335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.31.26.254:42181)
0927 18:45:30.394123 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:30.394136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:30.394178 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:45:30.395787 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:30.395811 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:30.396250 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:30.396613 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:30.398482 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:30.399947 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:30.401717 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:30.402426 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:30.402563 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:30.407614 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:30.407625 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:30.407676 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:30.407745 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:30.411033 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:30.411873 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:30.411881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:30.411889 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:30.412088 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:30.412560 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:30.412565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:30.412567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:30.413260 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:30.413589 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:30.414093 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 18:45:30.414552 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":386,"threads_started":1}
I20260927 18:45:30.424060 7012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e00576785742d98da3518458c87156" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42181 } }
I20260927 18:45:30.424515 7012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:30.424635 7012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e00576785742d98da3518458c87156, State: Initialized, Role: FOLLOWER
I20260927 18:45:30.425347 7012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e00576785742d98da3518458c87156" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42181 } }
I20260927 18:45:30.425591 7012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:30.425756 7012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:30.425941 7012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:30.428515 7012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e00576785742d98da3518458c87156" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42181 } }
I20260927 18:45:30.429060 7012 leader_election.cc:302] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e00576785742d98da3518458c87156; no voters:
I20260927 18:45:30.429733 7012 leader_election.cc:288] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:30.429867 7017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:30.430605 7017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [term 1 LEADER]: Becoming Leader. State: Replica: 85e00576785742d98da3518458c87156, State: Running, Role: LEADER
I20260927 18:45:30.431123 7017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e00576785742d98da3518458c87156" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42181 } }
I20260927 18:45:30.431915 7012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:30.434299 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e00576785742d98da3518458c87156. Latest consensus state: current_term: 1 leader_uuid: "85e00576785742d98da3518458c87156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e00576785742d98da3518458c87156" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42181 } } }
I20260927 18:45:30.434828 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:30.435134 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e00576785742d98da3518458c87156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e00576785742d98da3518458c87156" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42181 } } }
I20260927 18:45:30.437197 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156 [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:30.437578 7025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:30.447643 7025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:30.456409 7025 catalog_manager.cc:1347] Generated new cluster ID: 038050cb7ae642098bff400b0a382aef
I20260927 18:45:30.456549 7025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:30.478760 7025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:30.480777 7025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:30.488300 7025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Generated new TSK 0
I20260927 18:45:30.489490 7025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab.
Entry for principal oryx/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab.
I20260927 18:45:30.557181 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42181
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:30.908000 7038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:30.908365 7038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:30.908540 7038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:30.916219 7038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:30.916392 7038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:30.916538 7038 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:45:30.916673 7038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:30.926916 7038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:42181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:30.928297 7038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:30.929885 7038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:30.944154 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:30.944191 7046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:30.944219 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:30.945385 7038 server_base.cc:1034] running on GCE node
I20260927 18:45:30.946022 7038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:30.946991 7038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:30.948231 7038 hybrid_clock.cc:648] HybridClock initialized: now 1790534730948162 us; error 46 us; skew 500 ppm
I20260927 18:45:30.948558 7038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:30 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:30.951825 7038 init.cc:377] Logged in from keytab as oryx/127.31.26.193@KRBTEST.COM (short username oryx)
I20260927 18:45:30.953435 7038 webserver.cc:460] Webserver started at http://127.31.26.193:40535/ using document root <none> and password file <none>
I20260927 18:45:30.954408 7038 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:30.954528 7038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:30.954950 7038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:30.958081 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "72f541860ae04f3694eed63a55df0ebf"
format_stamp: "Formatted at 2026-09-27 18:45:30 on dist-test-slave-rfd5"
server_key: "738163dcad6323b94275eb81439e1111"
server_key_iv: "ce847a95fff9f1a1ee0c0ffac8c04729"
server_key_version: "encryptionkey@0"
I20260927 18:45:30.958921 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "72f541860ae04f3694eed63a55df0ebf"
format_stamp: "Formatted at 2026-09-27 18:45:30 on dist-test-slave-rfd5"
server_key: "738163dcad6323b94275eb81439e1111"
server_key_iv: "ce847a95fff9f1a1ee0c0ffac8c04729"
server_key_version: "encryptionkey@0"
I20260927 18:45:30.964322 7038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:45:30.968052 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:30.969528 7038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:30.969707 7038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "72f541860ae04f3694eed63a55df0ebf"
format_stamp: "Formatted at 2026-09-27 18:45:30 on dist-test-slave-rfd5"
server_key: "738163dcad6323b94275eb81439e1111"
server_key_iv: "ce847a95fff9f1a1ee0c0ffac8c04729"
server_key_version: "encryptionkey@0"
I20260927 18:45:30.969939 7038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:30.983392 7038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:30.987541 7038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:30.987829 7038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:30.990527 7038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:30.990660 7038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:30.990832 7038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:30.990913 7038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:45:31 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:31.017372 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:30.991121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:57629 (local address 127.31.26.254:42181)
0927 18:45:30.991960 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:30.991968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:30.992953 (+ 985us) server_negotiation.cc:407] Connection header received
0927 18:45:30.995049 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:30.995055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:30.995149 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:30.995271 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:30.997825 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:30.998939 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.001731 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.002320 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.002498 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.008117 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:31.008125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:31.008132 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:31.008181 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:31.012468 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.013821 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.013828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.013833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.014031 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.014809 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.014816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.014819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.015303 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:31.015505 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:31.016703 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0927 18:45:31.016958 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":633,"threads_started":1}
I20260927 18:45:31.018649 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:30.991796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.31.26.193:57629)
0927 18:45:30.992751 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:30.992810 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:30.994753 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:30.995574 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:30.995593 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:30.996505 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:30.997583 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:30.997624 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:30.999108 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:30.999114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.001480 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.001499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.002629 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.002642 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.003047 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.004044 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:31.004069 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:31.007686 (+ 3617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:31.012693 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.012705 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.012733 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.013641 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.014194 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.014200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.014203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.014564 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.015510 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:31.015522 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:31.016132 (+ 610us) client_negotiation.cc:769] Sending connection context
0927 18:45:31.016591 (+ 459us) client_negotiation.cc:240] Negotiation successful
0927 18:45:31.017269 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":421,"threads_started":1}
I20260927 18:45:31.040364 7038 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:42843
I20260927 18:45:31.040405 7173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:42843 every 8 connection(s)
I20260927 18:45:31.042024 7038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:31.042816 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:31.026096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:54179 (local address 127.31.26.254:42181)
0927 18:45:31.026294 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:31.026300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:31.027402 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 18:45:31.027616 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:31.027620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:31.027686 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:31.027812 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:31.029296 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.030476 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.031930 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.032732 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.032903 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.036056 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:31.036065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:31.036071 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:31.036116 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:31.039680 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.040632 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.040636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.040639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.040820 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.041278 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.041281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.041283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.041563 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:31.041674 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:31.042214 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 18:45:31.042451 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:45:31.042862 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 7038
I20260927 18:45:31.043045 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:31.043071 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:31.026289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.31.26.193:54179)
0927 18:45:31.027255 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:31.027283 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:31.027483 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:31.027851 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:31.027856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:31.028563 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:31.029061 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.029110 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.030648 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.030654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.031773 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.031787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.033008 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.033017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.033298 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.034157 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:31.034193 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:31.035802 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:31.039855 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.039862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.039866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.040428 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.040955 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.040960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.040964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.041166 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.041694 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:31.041699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:31.041903 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 18:45:31.042065 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:45:31.042424 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":501,"threads_started":1}
I20260927 18:45:31.043901 31851 external_mini_cluster.cc:1428] Setting key 59ab49f687490993685fc1ab69b43b3b
I20260927 18:45:31.045397 7174 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42181
I20260927 18:45:31.045658 7174 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:31.046520 7174 heartbeater.cc:511] Master 127.31.26.254:42181 requested a full tablet report, sending...
I20260927 18:45:31.048961 6965 ts_manager.cc:194] Registered new tserver with Master: 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843)
I20260927 18:45:31.051143 6965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:57629
WARNING: no policy specified for oryx/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab.
Entry for principal oryx/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab.
I20260927 18:45:31.096134 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42181
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:31.427039 7178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:31.427336 7178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:31.427496 7178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:31.435045 7178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:31.435209 7178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:31.435364 7178 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:45:31.435499 7178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:31.445797 7178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:42181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:31.447162 7178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:31.448559 7178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:31.462064 7183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:31.462105 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:31.462013 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:31.463102 7178 server_base.cc:1034] running on GCE node
I20260927 18:45:31.463717 7178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:31.464674 7178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:31.465889 7178 hybrid_clock.cc:648] HybridClock initialized: now 1790534731465830 us; error 32 us; skew 500 ppm
I20260927 18:45:31.466207 7178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:31 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534731, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:31.469554 7178 init.cc:377] Logged in from keytab as oryx/127.31.26.194@KRBTEST.COM (short username oryx)
I20260927 18:45:31.471097 7178 webserver.cc:460] Webserver started at http://127.31.26.194:37423/ using document root <none> and password file <none>
I20260927 18:45:31.471843 7178 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:31.471933 7178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:31.472308 7178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:31.475471 7178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "11602e61659546a78476b381cf8ef02e"
format_stamp: "Formatted at 2026-09-27 18:45:31 on dist-test-slave-rfd5"
server_key: "3160866297e7eb97e7979935be3c042e"
server_key_iv: "6dec656013b3f44a75e79cbcbd6fb2ed"
server_key_version: "encryptionkey@0"
I20260927 18:45:31.476444 7178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "11602e61659546a78476b381cf8ef02e"
format_stamp: "Formatted at 2026-09-27 18:45:31 on dist-test-slave-rfd5"
server_key: "3160866297e7eb97e7979935be3c042e"
server_key_iv: "6dec656013b3f44a75e79cbcbd6fb2ed"
server_key_version: "encryptionkey@0"
I20260927 18:45:31.482090 7178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:45:31.485857 7193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:31.487329 7178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:31.487525 7178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "11602e61659546a78476b381cf8ef02e"
format_stamp: "Formatted at 2026-09-27 18:45:31 on dist-test-slave-rfd5"
server_key: "3160866297e7eb97e7979935be3c042e"
server_key_iv: "6dec656013b3f44a75e79cbcbd6fb2ed"
server_key_version: "encryptionkey@0"
I20260927 18:45:31.487787 7178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:31.507294 7178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:31.512022 7178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:31.512318 7178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:31.515208 7178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:31.515328 7178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:31.515482 7178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:31.515576 7178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:45:31 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534731, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:31.545285 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:31.515770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:57557 (local address 127.31.26.254:42181)
0927 18:45:31.516002 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:31.516009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:31.517527 (+ 1518us) server_negotiation.cc:407] Connection header received
0927 18:45:31.520648 (+ 3121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:31.520654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:31.520770 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:31.520874 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:31.524121 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.525171 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.528608 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.529612 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.529762 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.536213 (+ 6451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:31.536224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:31.536233 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:31.536290 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:31.540729 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.541855 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.541865 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.541871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.542075 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.542579 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.542587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.542593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.543073 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:31.543268 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:31.544400 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0927 18:45:31.544724 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:45:31.546380 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:31.516403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.31.26.194:57557)
0927 18:45:31.517322 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:31.517381 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:31.520233 (+ 2852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:31.521563 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:31.521577 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:31.522621 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:31.523858 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.523895 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.525454 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.525459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.528347 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.528371 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.530271 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.530282 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.530757 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.531972 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:31.532026 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:31.535867 (+ 3841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:31.540971 (+ 5104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.540979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.540995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.541659 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.542232 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.542236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.542238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.542450 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.543278 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:31.543290 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:31.543904 (+ 614us) client_negotiation.cc:769] Sending connection context
0927 18:45:31.544196 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 18:45:31.544744 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":357,"threads_started":1}
I20260927 18:45:31.570000 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:31.552232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:47733 (local address 127.31.26.254:42181)
0927 18:45:31.552481 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:31.552491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:31.553331 (+ 840us) server_negotiation.cc:407] Connection header received
0927 18:45:31.553513 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:31.553516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:31.553579 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:31.553716 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:31.555389 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.556308 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.558574 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.559101 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.559251 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.562196 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:31.562202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:31.562206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:31.562240 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:31.565719 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.566820 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.566829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.566836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.567066 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:31.567764 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:31.567773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:31.567779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:31.568595 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:31.568783 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:31.569567 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 18:45:31.569757 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:45:31.570623 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:31.552194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.31.26.194:47733)
0927 18:45:31.553196 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:31.553224 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:31.553393 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:31.553730 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:31.553734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:31.554606 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:31.555157 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.555178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.556463 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.556468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.558407 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:31.558420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:31.559490 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:31.559494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:31.559753 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:31.560543 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:31.560587 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:31.561897 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:31.565879 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.565884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.565888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.566573 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.567244 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:31.567249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:31.567256 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:31.567468 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:31.568860 (+ 1392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:31.568868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:31.569215 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 18:45:31.569405 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 18:45:31.569940 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":396,"threads_started":1}
I20260927 18:45:31.576236 7178 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:36845
I20260927 18:45:31.576328 7312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:36845 every 8 connection(s)
I20260927 18:45:31.577589 7178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:31.579845 7313 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42181
I20260927 18:45:31.580138 7313 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:31.580832 7313 heartbeater.cc:511] Master 127.31.26.254:42181 requested a full tablet report, sending...
I20260927 18:45:31.582333 6965 ts_manager.cc:194] Registered new tserver with Master: 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845)
I20260927 18:45:31.583195 6965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:57557
I20260927 18:45:31.587773 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 7178
I20260927 18:45:31.587911 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:31.588402 31851 external_mini_cluster.cc:1428] Setting key 1b4aac48bdcdc1bdcdbdb31f94162e04
WARNING: no policy specified for oryx/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab.
Entry for principal oryx/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab.
I20260927 18:45:31.636838 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42181
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:31.968420 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:31.968794 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:31.968976 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:31.976478 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:31.976643 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:31.976790 7317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:45:31.976926 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:31.986869 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.26.254:42181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:31.988231 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:31.989748 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:32.004092 7325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:32.004161 7322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:32.004138 7323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:32.005313 7317 server_base.cc:1034] running on GCE node
I20260927 18:45:32.006074 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:32.007157 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:32.008427 7317 hybrid_clock.cc:648] HybridClock initialized: now 1790534732008355 us; error 47 us; skew 500 ppm
I20260927 18:45:32.008798 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:32 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534732, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:32.012010 7317 init.cc:377] Logged in from keytab as oryx/127.31.26.195@KRBTEST.COM (short username oryx)
I20260927 18:45:32.013494 7317 webserver.cc:460] Webserver started at http://127.31.26.195:38373/ using document root <none> and password file <none>
I20260927 18:45:32.014230 7317 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:32.014321 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:32.014602 7317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:32.017648 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "add65bcb91c64ea7951a8f8a2edd9763"
format_stamp: "Formatted at 2026-09-27 18:45:32 on dist-test-slave-rfd5"
server_key: "248b78fc76dc86a565ed716e1724ce69"
server_key_iv: "49b841de95a9afc75a10e9a6806393e4"
server_key_version: "encryptionkey@0"
I20260927 18:45:32.018391 7317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "add65bcb91c64ea7951a8f8a2edd9763"
format_stamp: "Formatted at 2026-09-27 18:45:32 on dist-test-slave-rfd5"
server_key: "248b78fc76dc86a565ed716e1724ce69"
server_key_iv: "49b841de95a9afc75a10e9a6806393e4"
server_key_version: "encryptionkey@0"
I20260927 18:45:32.023912 7317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:45:32.027679 7332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:32.029070 7317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:45:32.029280 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "add65bcb91c64ea7951a8f8a2edd9763"
format_stamp: "Formatted at 2026-09-27 18:45:32 on dist-test-slave-rfd5"
server_key: "248b78fc76dc86a565ed716e1724ce69"
server_key_iv: "49b841de95a9afc75a10e9a6806393e4"
server_key_version: "encryptionkey@0"
I20260927 18:45:32.029601 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:32.047515 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:32.051769 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:32.051995 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:32.054435 7174 heartbeater.cc:503] Master 127.31.26.254:42181 was elected leader, sending a full tablet report...
I20260927 18:45:32.054674 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:32.054780 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:32.054917 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:32.054992 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:45:32 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534732, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:32.079334 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:32.055336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:37373 (local address 127.31.26.254:42181)
0927 18:45:32.055501 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:32.055507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:32.057031 (+ 1524us) server_negotiation.cc:407] Connection header received
0927 18:45:32.059106 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:32.059113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:32.059199 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:32.059322 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:32.061343 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.062882 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.064720 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.065620 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.065782 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:32.071353 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:32.071362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:32.071370 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:32.071428 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:32.075145 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:32.076413 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:32.076419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:32.076423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:32.076560 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:32.077322 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:32.077329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:32.077335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:32.077710 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:32.077881 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:32.078726 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 18:45:32.078961 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:45:32.080466 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:32.055947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.31.26.195:37373)
0927 18:45:32.056878 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:32.056922 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:32.058825 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:32.059567 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:32.059575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:32.060106 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:32.061063 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:32.061136 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.063103 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.063107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:32.064544 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:32.064559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.065888 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.065894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:32.066122 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:32.067155 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:32.067194 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:32.071007 (+ 3813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:32.075438 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:32.075455 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:32.075494 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:32.076217 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:32.076751 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:32.076760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:32.076766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:32.077123 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:32.077944 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:32.077951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:32.078312 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 18:45:32.078584 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 18:45:32.079145 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":404,"threads_started":1}
I20260927 18:45:32.103227 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:32.087657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.31.26.195:54969)
0927 18:45:32.088294 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:32.088322 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:32.088491 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:32.088828 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:32.088833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:32.089379 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:32.089854 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:32.089869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.091120 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.091126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:32.092336 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:32.092353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.093838 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.093850 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:32.094107 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:32.095015 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:32.095049 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:32.096404 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:32.099864 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:32.099873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:32.099879 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:32.100587 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:32.101002 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:32.101010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:32.101014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:32.101222 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:32.101901 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:32.101906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:32.102193 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 18:45:32.102407 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 18:45:32.102791 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":360,"threads_started":1}
I20260927 18:45:32.103660 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:32.087618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:54969 (local address 127.31.26.254:42181)
0927 18:45:32.087819 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:32.087828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:32.088418 (+ 590us) server_negotiation.cc:407] Connection header received
0927 18:45:32.088614 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:32.088620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:32.088692 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:32.088802 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:32.090082 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.090974 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.092529 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.093363 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.093527 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:32.096625 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:32.096631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:32.096634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:32.096667 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:32.099705 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:32.100716 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:32.100720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:32.100723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:32.100859 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:32.101342 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:32.101350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:32.101355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:32.101781 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:32.101919 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:32.102829 (+ 910us) server_negotiation.cc:299] Negotiation successful
0927 18:45:32.103133 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:45:32.117237 7317 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:40057
I20260927 18:45:32.117295 7451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:40057 every 8 connection(s)
I20260927 18:45:32.118716 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:32.121443 7452 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42181
I20260927 18:45:32.121711 7452 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:32.122471 7452 heartbeater.cc:511] Master 127.31.26.254:42181 requested a full tablet report, sending...
I20260927 18:45:32.123878 6965 ts_manager.cc:194] Registered new tserver with Master: add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:32.124716 6965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:37373
I20260927 18:45:32.129139 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 7317
I20260927 18:45:32.129336 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:32.129755 31851 external_mini_cluster.cc:1428] Setting key 0ea152d65cf6ac8f4fc75b443d0ee443
I20260927 18:45:32.133090 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:45:32 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534732, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:45:32 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:32 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:32.585666 7313 heartbeater.cc:503] Master 127.31.26.254:42181 was elected leader, sending a full tablet report...
I20260927 18:45:33.127223 7452 heartbeater.cc:503] Master 127.31.26.254:42181 was elected leader, sending a full tablet report...
W20260927 18:45:33.248920 7460 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.068s user 0.000s sys 0.015s
W20260927 18:45:33.250250 7062 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:32.174100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.31.26.254:42181)
0927 18:45:32.174369 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:32.174376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:32.174694 (+ 318us) server_negotiation.cc:407] Connection header received
0927 18:45:32.174872 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:32.174878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:32.174941 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:32.175031 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:32.176384 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.177139 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.178987 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:32.179446 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:32.179590 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.249997 (+1070407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60092: BlockingRecv error: recv got EOF from 127.0.0.1:60092 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 18:45:33 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534732, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:33.272266 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.254176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.31.26.254:42181)
0927 18:45:33.254380 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.254385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.254807 (+ 422us) server_negotiation.cc:407] Connection header received
0927 18:45:33.254977 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.254982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.255073 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.255164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:33.256463 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.257288 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.258484 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.258888 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.258986 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.267102 (+ 8116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:33.267109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:33.267112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:33.267153 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:33.269741 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:33.270492 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:33.270496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:33.270499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:33.270619 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:33.270951 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:33.270954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:33.270956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:33.271252 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:33.271460 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.271723 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.271898 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:45:33.276782 6965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:33.283855 6965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:33.308715 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.296362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.31.26.194:36845)
0927 18:45:33.298704 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.298710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.298740 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:45:33.298861 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.298869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.299140 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.300448 (+ 1308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.300696 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.302728 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.306086 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.307457 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.307600 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.307821 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.308235 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.308411 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2198,"thread_start_us":2059,"threads_started":1}
I20260927 18:45:33.308897 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.296379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:42843 (local address 127.0.0.1:39698)
0927 18:45:33.298690 (+ 2311us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.298709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.298888 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.299941 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.299948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.299985 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.300519 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.300534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.303063 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.303072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.306074 (+ 3002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.306088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.308007 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.308014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.308181 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.308212 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.308304 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.308438 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2194,"mutex_wait_us":29,"thread_start_us":2060,"threads_started":1}
I20260927 18:45:33.309275 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.296132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:36845 (local address 127.0.0.1:32990)
0927 18:45:33.297301 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.297349 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.297607 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.299453 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.299465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.299509 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.300154 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.300171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.302949 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.302958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.305917 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.305929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.307672 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.307689 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.307932 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.308008 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.308182 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.308821 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":442,"thread_start_us":655,"threads_started":1}
I20260927 18:45:33.309666 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.296865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.31.26.193:42843)
0927 18:45:33.298243 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.298255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.298794 (+ 539us) server_negotiation.cc:407] Connection header received
0927 18:45:33.299113 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.299124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.299427 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.299628 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.300770 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.302849 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.306265 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.307671 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.307803 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.308005 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.309185 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.309369 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":1066,"threads_started":1}
I20260927 18:45:33.313586 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.298847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:40057 (local address 127.0.0.1:59892)
0927 18:45:33.305550 (+ 6703us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.305569 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.305706 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.306462 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.306466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.306493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.306898 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.306910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.308764 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.308769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.310936 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.310953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.312801 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.312807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.312975 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.313008 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.313141 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.313280 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6579,"thread_start_us":6427,"threads_started":1}
I20260927 18:45:33.313651 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.299070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.31.26.195:40057)
0927 18:45:33.304265 (+ 5195us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.304272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.305849 (+ 1577us) server_negotiation.cc:407] Connection header received
0927 18:45:33.305957 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.305968 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.306254 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.306448 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.307091 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.308599 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.311116 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.312540 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.312693 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.312903 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.313207 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.313381 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5048,"thread_start_us":4921,"threads_started":1}
I20260927 18:45:33.313637 7104 tablet_service.cc:1467] Processing CreateTablet for tablet a48c285bfa2845a68853c4526afa93c0 (DEFAULT_TABLE table=test-table [id=0212b59b577345f28bcdb150db7e2605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:33.313714 7243 tablet_service.cc:1467] Processing CreateTablet for tablet a48c285bfa2845a68853c4526afa93c0 (DEFAULT_TABLE table=test-table [id=0212b59b577345f28bcdb150db7e2605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:33.315219 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48c285bfa2845a68853c4526afa93c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:33.316460 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48c285bfa2845a68853c4526afa93c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:33.317377 7382 tablet_service.cc:1467] Processing CreateTablet for tablet a48c285bfa2845a68853c4526afa93c0 (DEFAULT_TABLE table=test-table [id=0212b59b577345f28bcdb150db7e2605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:33.318485 7382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48c285bfa2845a68853c4526afa93c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:33.329666 7477 tablet_bootstrap.cc:492] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Bootstrap starting.
I20260927 18:45:33.333006 7479 tablet_bootstrap.cc:492] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Bootstrap starting.
I20260927 18:45:33.334383 7477 tablet_bootstrap.cc:654] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:33.335265 7478 tablet_bootstrap.cc:492] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Bootstrap starting.
I20260927 18:45:33.335917 7477 log.cc:824] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:33.337265 7479 tablet_bootstrap.cc:654] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:33.338773 7479 log.cc:824] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:33.338910 7478 tablet_bootstrap.cc:654] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:33.339906 7477 tablet_bootstrap.cc:492] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: No bootstrap required, opened a new log
I20260927 18:45:33.340298 7478 log.cc:824] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:33.340329 7477 ts_tablet_manager.cc:1397] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:45:33.343583 7479 tablet_bootstrap.cc:492] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: No bootstrap required, opened a new log
I20260927 18:45:33.343709 7478 tablet_bootstrap.cc:492] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: No bootstrap required, opened a new log
I20260927 18:45:33.343982 7479 ts_tablet_manager.cc:1397] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260927 18:45:33.344048 7478 ts_tablet_manager.cc:1397] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 18:45:33.351523 7477 raft_consensus.cc:348] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.352102 7477 raft_consensus.cc:374] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:33.352265 7477 raft_consensus.cc:729] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add65bcb91c64ea7951a8f8a2edd9763, State: Initialized, Role: FOLLOWER
I20260927 18:45:33.352387 7479 raft_consensus.cc:348] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.352859 7479 raft_consensus.cc:374] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:33.352957 7479 raft_consensus.cc:729] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Initialized, Role: FOLLOWER
I20260927 18:45:33.352959 7477 consensus_queue.cc:260] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.353814 7479 consensus_queue.cc:260] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.354909 7477 ts_tablet_manager.cc:1428] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 18:45:33.355463 7479 ts_tablet_manager.cc:1428] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 18:45:33.357664 7478 raft_consensus.cc:348] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.358206 7478 raft_consensus.cc:374] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:33.358356 7478 raft_consensus.cc:729] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Initialized, Role: FOLLOWER
I20260927 18:45:33.359050 7478 consensus_queue.cc:260] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.361704 7478 ts_tablet_manager.cc:1428] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260927 18:45:33.371380 7453 tablet.cc:2369] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:33.494360 7484 raft_consensus.cc:482] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:33.494722 7484 raft_consensus.cc:504] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.496599 7484 leader_election.cc:288] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:33.503954 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.497874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:33865 (local address 127.31.26.195:40057)
0927 18:45:33.498073 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.498079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.498254 (+ 175us) server_negotiation.cc:407] Connection header received
0927 18:45:33.498363 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.498370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.498448 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.498545 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.499158 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.500009 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.502227 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.502990 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.503192 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.503270 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.503571 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.503701 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:45:33.503981 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.497148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:56155 (local address 127.31.26.193:42843)
0927 18:45:33.497377 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.497385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.497612 (+ 227us) server_negotiation.cc:407] Connection header received
0927 18:45:33.497877 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.497883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.497962 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.498028 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.498910 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.499965 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.502198 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.502992 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.503182 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.503272 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.503608 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.503738 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:45:33.504127 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.497543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:40057 (local address 127.31.26.194:33865)
0927 18:45:33.498147 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.498161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.498262 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.498577 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.498580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.498602 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.498993 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.499005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.500180 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.500184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.502053 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.502064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.503232 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.503239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.503404 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.503434 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.503545 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.503684 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":33,"thread_start_us":304,"threads_started":1}
I20260927 18:45:33.504371 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.496801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:42843 (local address 127.31.26.194:56155)
0927 18:45:33.497459 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.497500 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.497676 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.498286 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.498293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.498324 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.498754 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.498767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.500106 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.500114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.502033 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.502046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.503268 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.503274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.503435 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.503457 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.503573 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.503700 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":39,"thread_start_us":339,"threads_started":1}
I20260927 18:45:33.505191 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48c285bfa2845a68853c4526afa93c0" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763" is_pre_election: true
I20260927 18:45:33.505484 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48c285bfa2845a68853c4526afa93c0" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f541860ae04f3694eed63a55df0ebf" is_pre_election: true
I20260927 18:45:33.505805 7407 raft_consensus.cc:2454] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 0.
I20260927 18:45:33.505960 7124 raft_consensus.cc:2454] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 0.
I20260927 18:45:33.506562 7194 leader_election.cc:302] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, add65bcb91c64ea7951a8f8a2edd9763; no voters:
I20260927 18:45:33.506968 7484 raft_consensus.cc:2781] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:33.507202 7484 raft_consensus.cc:482] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:33.507340 7484 raft_consensus.cc:3037] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:33.509584 7484 raft_consensus.cc:504] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.510157 7484 leader_election.cc:288] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 election: Requested vote from peers 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:33.510731 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48c285bfa2845a68853c4526afa93c0" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f541860ae04f3694eed63a55df0ebf"
I20260927 18:45:33.510818 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48c285bfa2845a68853c4526afa93c0" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763"
I20260927 18:45:33.510905 7124 raft_consensus.cc:3037] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:33.511068 7407 raft_consensus.cc:3037] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:33.513039 7124 raft_consensus.cc:2454] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 1.
I20260927 18:45:33.513273 7407 raft_consensus.cc:2454] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 1.
I20260927 18:45:33.513551 7194 leader_election.cc:302] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, 72f541860ae04f3694eed63a55df0ebf; no voters:
I20260927 18:45:33.513939 7484 raft_consensus.cc:2781] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:33.514493 7484 raft_consensus.cc:686] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 1 LEADER]: Becoming Leader. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Running, Role: LEADER
I20260927 18:45:33.515079 7484 consensus_queue.cc:237] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.519574 6965 catalog_manager.cc:5495] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e reported cstate change: term changed from 0 to 1, leader changed from <none> to 11602e61659546a78476b381cf8ef02e (127.31.26.194). New cstate: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } health_report { overall_health: UNKNOWN } } }
W20260927 18:45:33.546365 7175 tablet.cc:2369] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:33.559271 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.554022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.31.26.194:36845)
0927 18:45:33.554240 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.554245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.554265 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:45:33.554351 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.554355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.554427 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.554558 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:33.554856 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.555552 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.557057 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.557543 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.557648 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.558161 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:33.558737 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:33.558856 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.558977 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.559051 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:45:33.568446 7407 raft_consensus.cc:1261] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Refusing update from remote peer 11602e61659546a78476b381cf8ef02e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:33.568588 7124 raft_consensus.cc:1261] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Refusing update from remote peer 11602e61659546a78476b381cf8ef02e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:33.569593 7484 consensus_queue.cc:1035] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [LEADER]: Connected to new peer: Peer: permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:33.569962 7488 consensus_queue.cc:1035] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:33.583665 7492 mvcc.cc:205] Tried to move back new op lower bound from 7334030268684996608 to 7334030268481380352. Current Snapshot: MvccSnapshot[applied={T|T < 7334030268684996608}]
I20260927 18:45:33.592590 7493 mvcc.cc:205] Tried to move back new op lower bound from 7334030268684996608 to 7334030268481380352. Current Snapshot: MvccSnapshot[applied={T|T < 7334030268684996608}]
I20260927 18:45:33.606570 6965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:33.606860 6965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:33.610419 6965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Sending DeleteTablet for 3 replicas of tablet a48c285bfa2845a68853c4526afa93c0
I20260927 18:45:33.611632 7243 tablet_service.cc:1514] Processing DeleteTablet for tablet a48c285bfa2845a68853c4526afa93c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:33 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:32990
I20260927 18:45:33.612038 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet a48c285bfa2845a68853c4526afa93c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:33 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:39698
I20260927 18:45:33.612285 7382 tablet_service.cc:1514] Processing DeleteTablet for tablet a48c285bfa2845a68853c4526afa93c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:33 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:59892
I20260927 18:45:33.614264 6965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:33.615425 6965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:33.623445 7500 tablet_replica.cc:330] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: stopping tablet replica
I20260927 18:45:33.623723 7501 tablet_replica.cc:330] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: stopping tablet replica
I20260927 18:45:33.623943 7500 raft_consensus.cc:2229] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:33.624318 7501 raft_consensus.cc:2229] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:33.624650 7500 raft_consensus.cc:2258] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:33.624943 7501 raft_consensus.cc:2258] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:33.626377 7243 tablet_service.cc:1467] Processing CreateTablet for tablet 72725d7590f94aac820a9e29f34ab7ea (DEFAULT_TABLE table=test-table [id=9b0adcf09b414635a9d566db49cd3b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:33.626698 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 72725d7590f94aac820a9e29f34ab7ea (DEFAULT_TABLE table=test-table [id=9b0adcf09b414635a9d566db49cd3b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:33.626891 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72725d7590f94aac820a9e29f34ab7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:33.627121 7500 ts_tablet_manager.cc:1905] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:33.627310 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72725d7590f94aac820a9e29f34ab7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:33.628476 7502 tablet_replica.cc:330] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: stopping tablet replica
I20260927 18:45:33.628743 7104 tablet_service.cc:1467] Processing CreateTablet for tablet 72725d7590f94aac820a9e29f34ab7ea (DEFAULT_TABLE table=test-table [id=9b0adcf09b414635a9d566db49cd3b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:33.629192 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72725d7590f94aac820a9e29f34ab7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:33.629876 7501 ts_tablet_manager.cc:1905] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:33.634573 7479 tablet_bootstrap.cc:492] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Bootstrap starting.
I20260927 18:45:33.635906 7478 tablet_bootstrap.cc:492] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Bootstrap starting.
I20260927 18:45:33.636261 7479 tablet_bootstrap.cc:654] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:33.637593 7478 tablet_bootstrap.cc:654] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:33.636034 7477 tablet_bootstrap.cc:492] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Bootstrap starting.
I20260927 18:45:33.639696 7501 ts_tablet_manager.cc:1918] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:33.639900 7501 log.cc:1196] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/a48c285bfa2845a68853c4526afa93c0
I20260927 18:45:33.640134 7477 tablet_bootstrap.cc:654] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:33.640556 7501 ts_tablet_manager.cc:1939] T a48c285bfa2845a68853c4526afa93c0 P 72f541860ae04f3694eed63a55df0ebf: Deleting consensus metadata
I20260927 18:45:33.643038 6951 catalog_manager.cc:4841] TS 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843): tablet a48c285bfa2845a68853c4526afa93c0 (table test-table [id=0212b59b577345f28bcdb150db7e2605]) successfully deleted
I20260927 18:45:33.645527 7502 raft_consensus.cc:2229] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:33.645615 7478 tablet_bootstrap.cc:492] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: No bootstrap required, opened a new log
I20260927 18:45:33.645830 7478 ts_tablet_manager.cc:1397] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 18:45:33.647552 7478 raft_consensus.cc:348] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.647868 7478 raft_consensus.cc:374] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:33.647857 7502 raft_consensus.cc:2258] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:33.647053 7477 tablet_bootstrap.cc:492] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: No bootstrap required, opened a new log
I20260927 18:45:33.648010 7478 raft_consensus.cc:729] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Initialized, Role: FOLLOWER
I20260927 18:45:33.648069 7477 ts_tablet_manager.cc:1397] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260927 18:45:33.648362 7478 consensus_queue.cc:260] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.649108 7477 raft_consensus.cc:348] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.649336 7477 raft_consensus.cc:374] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:33.649444 7477 raft_consensus.cc:729] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add65bcb91c64ea7951a8f8a2edd9763, State: Initialized, Role: FOLLOWER
I20260927 18:45:33.649762 7477 consensus_queue.cc:260] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.650267 7477 ts_tablet_manager.cc:1428] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:33.651759 7478 ts_tablet_manager.cc:1428] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 18:45:33.652405 7479 tablet_bootstrap.cc:492] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: No bootstrap required, opened a new log
I20260927 18:45:33.652591 7479 ts_tablet_manager.cc:1397] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20260927 18:45:33.653505 7502 ts_tablet_manager.cc:1905] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:33.653558 7479 raft_consensus.cc:348] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.653803 7479 raft_consensus.cc:374] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:33.653908 7479 raft_consensus.cc:729] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Initialized, Role: FOLLOWER
I20260927 18:45:33.654181 7479 consensus_queue.cc:260] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.654790 7479 ts_tablet_manager.cc:1428] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:45:33.660233 7500 ts_tablet_manager.cc:1918] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:33.660424 7500 log.cc:1196] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/a48c285bfa2845a68853c4526afa93c0
I20260927 18:45:33.660984 7500 ts_tablet_manager.cc:1939] T a48c285bfa2845a68853c4526afa93c0 P 11602e61659546a78476b381cf8ef02e: Deleting consensus metadata
I20260927 18:45:33.662766 6953 catalog_manager.cc:4841] TS 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845): tablet a48c285bfa2845a68853c4526afa93c0 (table test-table [id=0212b59b577345f28bcdb150db7e2605]) successfully deleted
I20260927 18:45:33.663750 7502 ts_tablet_manager.cc:1918] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:33.663908 7502 log.cc:1196] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/a48c285bfa2845a68853c4526afa93c0
I20260927 18:45:33.664376 7502 ts_tablet_manager.cc:1939] T a48c285bfa2845a68853c4526afa93c0 P add65bcb91c64ea7951a8f8a2edd9763: Deleting consensus metadata
I20260927 18:45:33.665917 6951 catalog_manager.cc:4841] TS add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057): tablet a48c285bfa2845a68853c4526afa93c0 (table test-table [id=0212b59b577345f28bcdb150db7e2605]) successfully deleted
W20260927 18:45:33.834462 7314 tablet.cc:2369] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:45:33.978518 7485 raft_consensus.cc:482] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:33.978802 7485 raft_consensus.cc:504] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.980782 7485 leader_election.cc:288] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:33.989019 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.981702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:54945 (local address 127.31.26.195:40057)
0927 18:45:33.981866 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.981872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.982502 (+ 630us) server_negotiation.cc:407] Connection header received
0927 18:45:33.982805 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.982812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.982920 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.983012 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.983702 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.984553 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.986861 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.988025 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.988187 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.988259 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.988611 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.988749 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:45:33.989069 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.981061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:40057 (local address 127.31.26.193:54945)
0927 18:45:33.982387 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.982407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.982633 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.983054 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.983060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.983093 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.983523 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.983536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.984697 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.984703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.986683 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.986697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.988265 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.988272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.988454 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.988494 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.988588 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.988722 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":1000,"threads_started":1}
I20260927 18:45:33.989904 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.981250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:36845 (local address 127.31.26.193:39545)
0927 18:45:33.982794 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:33.982811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:33.982988 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:33.983485 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:33.983490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:33.983519 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:33.983894 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.983906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.985458 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.985463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.987210 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:33.987220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.988491 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.988497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:33.988675 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.988707 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:45:33.989494 (+ 787us) client_negotiation.cc:240] Negotiation successful
0927 18:45:33.989624 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"thread_start_us":1217,"threads_started":1}
I20260927 18:45:33.989801 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72725d7590f94aac820a9e29f34ab7ea" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763" is_pre_election: true
I20260927 18:45:33.990070 7407 raft_consensus.cc:2454] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 0.
I20260927 18:45:33.990411 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:33.981344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:39545 (local address 127.31.26.194:36845)
0927 18:45:33.981539 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:33.981546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:33.982912 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 18:45:33.983153 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:33.983159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:33.983260 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:33.983313 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:33.984079 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.985318 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.987365 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:33.988276 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:33.988434 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:33.988504 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:33.988849 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 18:45:33.988980 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:45:33.990557 7054 leader_election.cc:302] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f541860ae04f3694eed63a55df0ebf, add65bcb91c64ea7951a8f8a2edd9763; no voters:
I20260927 18:45:33.990990 7485 raft_consensus.cc:2781] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:33.991122 7485 raft_consensus.cc:482] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:33.991218 7485 raft_consensus.cc:3037] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:33.991307 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72725d7590f94aac820a9e29f34ab7ea" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11602e61659546a78476b381cf8ef02e" is_pre_election: true
I20260927 18:45:33.991773 7268 raft_consensus.cc:2454] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 0.
I20260927 18:45:33.993811 7485 raft_consensus.cc:504] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:33.994419 7485 leader_election.cc:288] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 election: Requested vote from peers 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:33.994863 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72725d7590f94aac820a9e29f34ab7ea" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11602e61659546a78476b381cf8ef02e"
I20260927 18:45:33.994944 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72725d7590f94aac820a9e29f34ab7ea" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763"
I20260927 18:45:33.995059 7268 raft_consensus.cc:3037] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:33.995160 7407 raft_consensus.cc:3037] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:33.997380 7268 raft_consensus.cc:2454] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 1.
I20260927 18:45:33.997825 7056 leader_election.cc:302] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, 72f541860ae04f3694eed63a55df0ebf; no voters:
I20260927 18:45:33.998087 7407 raft_consensus.cc:2454] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 1.
I20260927 18:45:33.998133 7485 raft_consensus.cc:2781] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:33.998715 7485 raft_consensus.cc:686] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 1 LEADER]: Becoming Leader. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Running, Role: LEADER
I20260927 18:45:33.999095 7485 consensus_queue.cc:237] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.002756 6967 catalog_manager.cc:5495] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 72f541860ae04f3694eed63a55df0ebf (127.31.26.193). New cstate: current_term: 1 leader_uuid: "72f541860ae04f3694eed63a55df0ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:45:34 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534730, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.254@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:34.042783 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.027548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42181 (local address 127.0.0.1:60114)
0927 18:45:34.028414 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:34.028431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:34.028606 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:34.028976 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:34.028985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:34.029698 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:34.029905 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.029914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.030970 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.030975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.032081 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.032095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.032861 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.032867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.033043 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.033928 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:34.033950 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:34.036245 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:34.040314 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:34.040327 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:34.040333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:34.040812 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:34.041186 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:34.041190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:34.041200 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:34.041339 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:34.041959 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:34.041965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:34.042114 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 18:45:34.042266 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:45:34.042494 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":601,"threads_started":1}
I20260927 18:45:34.043283 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.027533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.31.26.254:42181)
0927 18:45:34.028311 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.028317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.028524 (+ 207us) server_negotiation.cc:407] Connection header received
0927 18:45:34.028751 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.028754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.028817 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.028929 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:34.030052 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.030831 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.032216 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.032650 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.032850 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.036463 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:34.036471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:34.036476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:34.036516 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:34.040154 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:34.040920 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:34.040926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:34.040929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:34.041068 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:34.041444 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:34.041448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:34.041451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:34.041842 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:34.041984 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.042385 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.042667 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":605,"thread_start_us":335,"threads_started":1}
I20260927 18:45:34.048683 6967 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:60114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:45:34.049841 6967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:34.056877 7104 tablet_service.cc:1467] Processing CreateTablet for tablet daeffa3686c44bf18ec020db7081f531 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdd66a9fbf76459697e0ebc887021583]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:45:34.056897 7243 tablet_service.cc:1467] Processing CreateTablet for tablet daeffa3686c44bf18ec020db7081f531 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdd66a9fbf76459697e0ebc887021583]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:45:34.056996 7381 tablet_service.cc:1467] Processing CreateTablet for tablet daeffa3686c44bf18ec020db7081f531 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdd66a9fbf76459697e0ebc887021583]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:45:34.057466 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daeffa3686c44bf18ec020db7081f531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.057469 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daeffa3686c44bf18ec020db7081f531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.057487 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daeffa3686c44bf18ec020db7081f531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.063349 7478 tablet_bootstrap.cc:492] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: Bootstrap starting.
I20260927 18:45:34.065130 7478 tablet_bootstrap.cc:654] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.065302 7479 tablet_bootstrap.cc:492] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: Bootstrap starting.
I20260927 18:45:34.066934 7479 tablet_bootstrap.cc:654] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.067399 7478 tablet_bootstrap.cc:492] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: No bootstrap required, opened a new log
I20260927 18:45:34.067561 7478 ts_tablet_manager.cc:1397] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260927 18:45:34.067977 7477 tablet_bootstrap.cc:492] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: Bootstrap starting.
I20260927 18:45:34.068275 7478 raft_consensus.cc:348] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.068560 7478 raft_consensus.cc:374] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.068647 7478 raft_consensus.cc:729] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.069286 7478 consensus_queue.cc:260] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.069756 7477 tablet_bootstrap.cc:654] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.069752 7485 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.069975 7479 tablet_bootstrap.cc:492] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: No bootstrap required, opened a new log
I20260927 18:45:34.070037 7485 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.070133 7479 ts_tablet_manager.cc:1397] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 18:45:34.070470 7478 ts_tablet_manager.cc:1428] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 18:45:34.070883 7479 raft_consensus.cc:348] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.071147 7479 raft_consensus.cc:374] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.071251 7479 raft_consensus.cc:729] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.071506 7479 consensus_queue.cc:260] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.071946 7477 tablet_bootstrap.cc:492] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: No bootstrap required, opened a new log
I20260927 18:45:34.072038 7479 ts_tablet_manager.cc:1428] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:45:34.071897 7488 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.072098 7477 ts_tablet_manager.cc:1397] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:45:34.072190 7488 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.073709 7477 raft_consensus.cc:348] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.073982 7477 raft_consensus.cc:374] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.074064 7477 raft_consensus.cc:729] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add65bcb91c64ea7951a8f8a2edd9763, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.074353 7477 consensus_queue.cc:260] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.074975 7477 ts_tablet_manager.cc:1428] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:45:34.074872 7483 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.075259 7483 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.226257 7488 raft_consensus.cc:482] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:34.226455 7488 raft_consensus.cc:504] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.226980 7488 leader_election.cc:288] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:34.227684 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daeffa3686c44bf18ec020db7081f531" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763" is_pre_election: true
I20260927 18:45:34.227698 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daeffa3686c44bf18ec020db7081f531" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f541860ae04f3694eed63a55df0ebf" is_pre_election: true
I20260927 18:45:34.227941 7407 raft_consensus.cc:2454] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 0.
I20260927 18:45:34.227941 7123 raft_consensus.cc:2454] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 0.
I20260927 18:45:34.228377 7194 leader_election.cc:302] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, 72f541860ae04f3694eed63a55df0ebf; no voters:
I20260927 18:45:34.228664 7488 raft_consensus.cc:2781] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:34.228770 7488 raft_consensus.cc:482] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:34.228874 7488 raft_consensus.cc:3037] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:34.230829 7488 raft_consensus.cc:504] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.231376 7488 leader_election.cc:288] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 election: Requested vote from peers 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:34.231873 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daeffa3686c44bf18ec020db7081f531" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f541860ae04f3694eed63a55df0ebf"
I20260927 18:45:34.231914 7407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daeffa3686c44bf18ec020db7081f531" candidate_uuid: "11602e61659546a78476b381cf8ef02e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763"
I20260927 18:45:34.232077 7123 raft_consensus.cc:3037] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:34.232249 7407 raft_consensus.cc:3037] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:34.234401 7407 raft_consensus.cc:2454] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 1.
I20260927 18:45:34.234463 7123 raft_consensus.cc:2454] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11602e61659546a78476b381cf8ef02e in term 1.
I20260927 18:45:34.234870 7194 leader_election.cc:302] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, add65bcb91c64ea7951a8f8a2edd9763; no voters:
I20260927 18:45:34.235178 7488 raft_consensus.cc:2781] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:34.235600 7488 raft_consensus.cc:686] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [term 1 LEADER]: Becoming Leader. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Running, Role: LEADER
I20260927 18:45:34.235881 7488 consensus_queue.cc:237] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.237460 7520 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: TxnStatusTablet state changed. Reason: New leader 11602e61659546a78476b381cf8ef02e. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.237788 7520 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:45:34.238464 7522 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:45:34.238691 7522 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:45:34.239152 6966 catalog_manager.cc:5495] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e reported cstate change: term changed from 0 to 1, leader changed from <none> to 11602e61659546a78476b381cf8ef02e (127.31.26.194). New cstate: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260927 18:45:34.271546 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.265644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.31.26.194:36845)
0927 18:45:34.265861 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.265868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.266000 (+ 132us) server_negotiation.cc:407] Connection header received
0927 18:45:34.266286 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.266290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.266354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.266421 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:34.266853 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.267585 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.269642 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.270147 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.270256 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.270861 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.270974 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.271073 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.271243 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.271314 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:45:34.271726 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.265635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:36845 (local address 127.0.0.1:33008)
0927 18:45:34.265901 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:34.265926 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:34.266084 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:34.266480 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:34.266486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:34.266499 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:45:34.266707 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.266721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.267733 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.267741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.269459 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.269477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.270322 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.270330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.270569 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.270650 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.271101 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.271126 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:45:34.271220 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:45:34.271394 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:45:34.276343 7123 raft_consensus.cc:1261] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Refusing update from remote peer 11602e61659546a78476b381cf8ef02e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:34.276413 7407 raft_consensus.cc:1261] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Refusing update from remote peer 11602e61659546a78476b381cf8ef02e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:34.277048 7520 consensus_queue.cc:1035] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [LEADER]: Connected to new peer: Peer: permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:34.277415 7488 consensus_queue.cc:1035] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:34.284457 7483 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: TxnStatusTablet state changed. Reason: New leader 11602e61659546a78476b381cf8ef02e. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.284822 7483 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.286406 7525 mvcc.cc:205] Tried to move back new op lower bound from 7334030271589400576 to 7334030271432261632. Current Snapshot: MvccSnapshot[applied={T|T < 7334030271589400576}]
I20260927 18:45:34.289561 7483 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.289836 7483 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P add65bcb91c64ea7951a8f8a2edd9763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.290932 7485 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: TxnStatusTablet state changed. Reason: New leader 11602e61659546a78476b381cf8ef02e. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.291232 7485 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.292629 7488 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.294562 7488 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:45:34.295672 7488 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.295925 7488 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 11602e61659546a78476b381cf8ef02e: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:45:34.298666 7485 tablet_replica.cc:439] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11602e61659546a78476b381cf8ef02e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } } }
I20260927 18:45:34.298947 7485 tablet_replica.cc:442] T daeffa3686c44bf18ec020db7081f531 P 72f541860ae04f3694eed63a55df0ebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:45:34.309002 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.303090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.31.26.193:42843)
0927 18:45:34.303807 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.303814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.303832 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:45:34.303931 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.303934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.304014 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.304194 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:34.304641 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.305451 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.306782 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.307204 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.307340 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.307783 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.308352 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.308513 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.308593 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.308680 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":459,"threads_started":1}
I20260927 18:45:34.325421 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.319663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:55975 (local address 127.31.26.194:36845)
0927 18:45:34.319847 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.319852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.320391 (+ 539us) server_negotiation.cc:407] Connection header received
0927 18:45:34.320605 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.320609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.320683 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.320781 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:34.321369 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.322140 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.323535 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.323984 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.324136 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.324576 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.324730 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.324836 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.324974 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.325059 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:45:34.325507 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.319678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:36845 (local address 127.31.26.193:55975)
0927 18:45:34.320271 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:34.320287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:34.320462 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:34.320796 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:34.320801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:34.320813 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:45:34.321221 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.321231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.322281 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.322287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.323387 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.323398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.324185 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.324193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.324340 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.324394 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.324838 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.324852 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:45:34.324953 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:45:34.325130 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":307,"threads_started":1}
I20260927 18:45:34.336092 7407 raft_consensus.cc:1261] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Refusing update from remote peer 72f541860ae04f3694eed63a55df0ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:34.336166 7268 raft_consensus.cc:1261] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Refusing update from remote peer 72f541860ae04f3694eed63a55df0ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:34.336835 7508 consensus_queue.cc:1035] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:34.337337 7485 consensus_queue.cc:1035] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:34.376495 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.370282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:54869 (local address 127.31.26.193:42843)
0927 18:45:34.370464 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.370470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.370674 (+ 204us) server_negotiation.cc:407] Connection header received
0927 18:45:34.370854 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.370860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.370949 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.371048 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:34.371763 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.372619 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.374577 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.374999 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.375163 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.375674 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.375825 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.375924 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.376097 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.376186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:45:34.377012 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.369884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:42843 (local address 127.31.26.194:54869)
0927 18:45:34.370547 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:34.370570 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:34.370723 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:34.371108 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:34.371112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:34.371124 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:45:34.371523 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.371535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.372752 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.372763 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.374426 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.374444 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.375179 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.375186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.375359 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.375450 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.375951 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.375963 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:45:34.376056 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:45:34.376201 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":596,"threads_started":1}
I20260927 18:45:34.420495 6966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:34.423171 6966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:34.427191 6966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Sending DeleteTablet for 3 replicas of tablet 72725d7590f94aac820a9e29f34ab7ea
I20260927 18:45:34.428391 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 72725d7590f94aac820a9e29f34ab7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:34 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:59892
I20260927 18:45:34.428489 7243 tablet_service.cc:1514] Processing DeleteTablet for tablet 72725d7590f94aac820a9e29f34ab7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:34 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:32990
I20260927 18:45:34.428848 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet 72725d7590f94aac820a9e29f34ab7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:34 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:39698
I20260927 18:45:34.429324 7548 tablet_replica.cc:330] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: stopping tablet replica
I20260927 18:45:34.429525 7548 raft_consensus.cc:2229] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:34.429811 7548 raft_consensus.cc:2258] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:34.430948 7548 ts_tablet_manager.cc:1905] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:34.438709 7548 ts_tablet_manager.cc:1918] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:45:34.438884 7548 log.cc:1196] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/72725d7590f94aac820a9e29f34ab7ea
I20260927 18:45:34.439347 7548 ts_tablet_manager.cc:1939] T 72725d7590f94aac820a9e29f34ab7ea P add65bcb91c64ea7951a8f8a2edd9763: Deleting consensus metadata
I20260927 18:45:34.440531 6951 catalog_manager.cc:4841] TS add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057): tablet 72725d7590f94aac820a9e29f34ab7ea (table test-table [id=9b0adcf09b414635a9d566db49cd3b85]) successfully deleted
I20260927 18:45:34.443660 7549 tablet_replica.cc:330] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: stopping tablet replica
I20260927 18:45:34.443938 7549 raft_consensus.cc:2229] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:34.444664 7549 raft_consensus.cc:2258] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:34.445784 7549 ts_tablet_manager.cc:1905] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:34.453855 7549 ts_tablet_manager.cc:1918] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:45:34.454025 7549 log.cc:1196] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/72725d7590f94aac820a9e29f34ab7ea
I20260927 18:45:34.454530 7549 ts_tablet_manager.cc:1939] T 72725d7590f94aac820a9e29f34ab7ea P 72f541860ae04f3694eed63a55df0ebf: Deleting consensus metadata
I20260927 18:45:34.454559 7550 tablet_replica.cc:330] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: stopping tablet replica
I20260927 18:45:34.454977 7550 raft_consensus.cc:2229] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:34.455348 7550 raft_consensus.cc:2258] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:34.455825 6951 catalog_manager.cc:4841] TS 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843): tablet 72725d7590f94aac820a9e29f34ab7ea (table test-table [id=9b0adcf09b414635a9d566db49cd3b85]) successfully deleted
I20260927 18:45:34.456878 7550 ts_tablet_manager.cc:1905] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:34.465559 7550 ts_tablet_manager.cc:1918] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:45:34.465749 7550 log.cc:1196] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/72725d7590f94aac820a9e29f34ab7ea
I20260927 18:45:34.466248 7550 ts_tablet_manager.cc:1939] T 72725d7590f94aac820a9e29f34ab7ea P 11602e61659546a78476b381cf8ef02e: Deleting consensus metadata
I20260927 18:45:34.467525 6953 catalog_manager.cc:4841] TS 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845): tablet 72725d7590f94aac820a9e29f34ab7ea (table test-table [id=9b0adcf09b414635a9d566db49cd3b85]) successfully deleted
Sep 27 18:45:34 dist-test-slave-rfd5 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:45:34 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:34 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:45:34.504438 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:34.493363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.31.26.254:42181)
0927 18:45:34.493639 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.493650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.494197 (+ 547us) server_negotiation.cc:407] Connection header received
0927 18:45:34.494419 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.494425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.494505 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.494613 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:34.495995 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.496748 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.498110 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.498537 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.498689 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.504252 (+ 5563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60126: BlockingRecv error: recv got EOF from 127.0.0.1:60126 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 18:45:34 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM
I20260927 18:45:34.522897 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.508514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.31.26.254:42181)
0927 18:45:34.508705 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.508712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.508732 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:45:34.508904 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.508912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.508995 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.509056 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:34.510434 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.511165 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.512555 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.513190 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.513328 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.517453 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:34.517458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:34.517461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:34.517499 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:34.520085 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:34.520798 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:34.520806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:34.520811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:34.520958 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:34.521447 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:34.521455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:34.521463 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:34.521718 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:34.521989 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.522289 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.522518 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:45:34.526619 6966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:45:34.527259 6966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:34.533773 7104 tablet_service.cc:1467] Processing CreateTablet for tablet 870cbc3e6a15412a9fde5a66217a4ed4 (DEFAULT_TABLE table=test-table [id=5566508dedeb4b08a6156b1b9122ba49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:34.534260 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870cbc3e6a15412a9fde5a66217a4ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.534350 7243 tablet_service.cc:1467] Processing CreateTablet for tablet 870cbc3e6a15412a9fde5a66217a4ed4 (DEFAULT_TABLE table=test-table [id=5566508dedeb4b08a6156b1b9122ba49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:34.534781 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870cbc3e6a15412a9fde5a66217a4ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.538004 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 870cbc3e6a15412a9fde5a66217a4ed4 (DEFAULT_TABLE table=test-table [id=5566508dedeb4b08a6156b1b9122ba49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:34.538473 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870cbc3e6a15412a9fde5a66217a4ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.541561 7478 tablet_bootstrap.cc:492] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Bootstrap starting.
I20260927 18:45:34.543467 7478 tablet_bootstrap.cc:654] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.556211 7479 tablet_bootstrap.cc:492] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Bootstrap starting.
I20260927 18:45:34.558064 7479 tablet_bootstrap.cc:654] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.559212 7477 tablet_bootstrap.cc:492] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Bootstrap starting.
I20260927 18:45:34.560472 7478 tablet_bootstrap.cc:492] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: No bootstrap required, opened a new log
I20260927 18:45:34.560693 7478 ts_tablet_manager.cc:1397] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260927 18:45:34.560815 7477 tablet_bootstrap.cc:654] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.561606 7478 raft_consensus.cc:348] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.562199 7478 raft_consensus.cc:374] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.562306 7478 raft_consensus.cc:729] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.562595 7478 consensus_queue.cc:260] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.563112 7477 tablet_bootstrap.cc:492] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: No bootstrap required, opened a new log
I20260927 18:45:34.563213 7478 ts_tablet_manager.cc:1428] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:45:34.563254 7477 ts_tablet_manager.cc:1397] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 18:45:34.563962 7477 raft_consensus.cc:348] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.564200 7477 raft_consensus.cc:374] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.564296 7477 raft_consensus.cc:729] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add65bcb91c64ea7951a8f8a2edd9763, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.565114 7479 tablet_bootstrap.cc:492] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: No bootstrap required, opened a new log
I20260927 18:45:34.565268 7479 ts_tablet_manager.cc:1397] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260927 18:45:34.566008 7479 raft_consensus.cc:348] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.566277 7479 raft_consensus.cc:374] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.566375 7479 raft_consensus.cc:729] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.566589 7479 consensus_queue.cc:260] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.567118 7479 ts_tablet_manager.cc:1428] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 18:45:34.567410 7477 consensus_queue.cc:260] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.567972 7477 ts_tablet_manager.cc:1428] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260927 18:45:34.697240 7483 raft_consensus.cc:482] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:34.697475 7483 raft_consensus.cc:504] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.699213 7483 leader_election.cc:288] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843), 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845)
I20260927 18:45:34.717711 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.700436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:50173 (local address 127.31.26.193:42843)
0927 18:45:34.700618 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.700624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.710198 (+ 9574us) server_negotiation.cc:407] Connection header received
0927 18:45:34.710715 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.710721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.710797 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.710856 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:34.711578 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.712670 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.715157 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.716473 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.716604 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.716687 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.717233 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.717376 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:45:34.718762 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.699471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:42843 (local address 127.31.26.195:50173)
0927 18:45:34.710049 (+ 10578us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:34.710074 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:34.710513 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:34.711061 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:34.711066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:34.711095 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:34.711379 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.711393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.712881 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.712886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.714982 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.714996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.716823 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.716829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.717008 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.717045 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:45:34.718194 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0927 18:45:34.718340 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10421,"thread_start_us":10751,"threads_started":1}
I20260927 18:45:34.719614 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870cbc3e6a15412a9fde5a66217a4ed4" candidate_uuid: "add65bcb91c64ea7951a8f8a2edd9763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f541860ae04f3694eed63a55df0ebf" is_pre_election: true
I20260927 18:45:34.719869 7123 raft_consensus.cc:2454] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add65bcb91c64ea7951a8f8a2edd9763 in term 0.
I20260927 18:45:34.720546 7333 leader_election.cc:302] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f541860ae04f3694eed63a55df0ebf, add65bcb91c64ea7951a8f8a2edd9763; no voters:
I20260927 18:45:34.721035 7483 raft_consensus.cc:2781] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:34.721189 7483 raft_consensus.cc:482] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:34.721282 7483 raft_consensus.cc:3037] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:34.722050 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.700110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:57465 (local address 127.31.26.194:36845)
0927 18:45:34.700287 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.700293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.711166 (+ 10873us) server_negotiation.cc:407] Connection header received
0927 18:45:34.714843 (+ 3677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.714848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.714944 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.715006 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:45:34.715770 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.716844 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.719356 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.720645 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.720794 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.720897 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.721568 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.721726 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:45:34.723031 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.699718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:36845 (local address 127.31.26.195:57465)
0927 18:45:34.711032 (+ 11314us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:34.711055 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:34.714645 (+ 3590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:34.715190 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:34.715194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:34.715218 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:45:34.715564 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.715574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.717031 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.717039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.719173 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:34.719189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.721156 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.721163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:34.721370 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.721405 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:45:34.722399 (+ 994us) client_negotiation.cc:240] Negotiation successful
0927 18:45:34.722598 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11183,"thread_start_us":11459,"threads_started":1}
I20260927 18:45:34.723650 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870cbc3e6a15412a9fde5a66217a4ed4" candidate_uuid: "add65bcb91c64ea7951a8f8a2edd9763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11602e61659546a78476b381cf8ef02e" is_pre_election: true
I20260927 18:45:34.723904 7268 raft_consensus.cc:2454] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add65bcb91c64ea7951a8f8a2edd9763 in term 0.
I20260927 18:45:34.724651 7483 raft_consensus.cc:504] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.725736 7483 leader_election.cc:288] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [CANDIDATE]: Term 1 election: Requested vote from peers 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843), 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845)
I20260927 18:45:34.726303 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870cbc3e6a15412a9fde5a66217a4ed4" candidate_uuid: "add65bcb91c64ea7951a8f8a2edd9763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f541860ae04f3694eed63a55df0ebf"
I20260927 18:45:34.726393 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870cbc3e6a15412a9fde5a66217a4ed4" candidate_uuid: "add65bcb91c64ea7951a8f8a2edd9763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11602e61659546a78476b381cf8ef02e"
I20260927 18:45:34.726536 7123 raft_consensus.cc:3037] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:34.726610 7268 raft_consensus.cc:3037] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:34.729979 7123 raft_consensus.cc:2454] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add65bcb91c64ea7951a8f8a2edd9763 in term 1.
I20260927 18:45:34.729986 7268 raft_consensus.cc:2454] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add65bcb91c64ea7951a8f8a2edd9763 in term 1.
I20260927 18:45:34.730522 7333 leader_election.cc:302] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, 72f541860ae04f3694eed63a55df0ebf, add65bcb91c64ea7951a8f8a2edd9763; no voters:
I20260927 18:45:34.730849 7483 raft_consensus.cc:2781] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:34.731354 7483 raft_consensus.cc:686] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 LEADER]: Becoming Leader. State: Replica: add65bcb91c64ea7951a8f8a2edd9763, State: Running, Role: LEADER
I20260927 18:45:34.731866 7483 consensus_queue.cc:237] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.735898 6966 catalog_manager.cc:5495] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 reported cstate change: term changed from 0 to 1, leader changed from <none> to add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195). New cstate: current_term: 1 leader_uuid: "add65bcb91c64ea7951a8f8a2edd9763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } health_report { overall_health: HEALTHY } } }
I20260927 18:45:34.763458 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:34.756736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.31.26.195:40057)
0927 18:45:34.757332 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:34.757338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:34.757361 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:45:34.757468 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:34.757474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:34.757600 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:34.757786 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:34.758132 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.759233 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.760524 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:34.761184 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:34.761317 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:34.761767 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:34.762524 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:34.762687 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:34.762805 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 18:45:34.762891 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
I20260927 18:45:34.767467 7123 raft_consensus.cc:1261] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Refusing update from remote peer add65bcb91c64ea7951a8f8a2edd9763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:34.767537 7268 raft_consensus.cc:1261] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Refusing update from remote peer add65bcb91c64ea7951a8f8a2edd9763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:34.768262 7568 consensus_queue.cc:1035] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:34.768663 7483 consensus_queue.cc:1035] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:34.788038 6967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:34.788319 6967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:34.791764 6967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Sending DeleteTablet for 3 replicas of tablet 870cbc3e6a15412a9fde5a66217a4ed4
I20260927 18:45:34.792752 7243 tablet_service.cc:1514] Processing DeleteTablet for tablet 870cbc3e6a15412a9fde5a66217a4ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:34 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:32990
I20260927 18:45:34.792747 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 870cbc3e6a15412a9fde5a66217a4ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:34 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:59892
I20260927 18:45:34.793037 7550 tablet_replica.cc:330] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: stopping tablet replica
I20260927 18:45:34.793046 7548 tablet_replica.cc:330] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: stopping tablet replica
I20260927 18:45:34.793246 7550 raft_consensus.cc:2229] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:34.793275 7548 raft_consensus.cc:2229] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:34.793327 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet 870cbc3e6a15412a9fde5a66217a4ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:34 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:39698
I20260927 18:45:34.793474 7550 raft_consensus.cc:2258] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:34.793828 7548 raft_consensus.cc:2258] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:34.794130 7549 tablet_replica.cc:330] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: stopping tablet replica
I20260927 18:45:34.794323 7549 raft_consensus.cc:2229] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:34.794523 7549 raft_consensus.cc:2258] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:34.794710 7550 ts_tablet_manager.cc:1905] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:34.795615 7548 ts_tablet_manager.cc:1905] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:34.795671 7549 ts_tablet_manager.cc:1905] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:34.801183 7550 ts_tablet_manager.cc:1918] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:34.794553 6967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:45:34.801368 7550 log.cc:1196] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/870cbc3e6a15412a9fde5a66217a4ed4
I20260927 18:45:34.801793 7550 ts_tablet_manager.cc:1939] T 870cbc3e6a15412a9fde5a66217a4ed4 P 11602e61659546a78476b381cf8ef02e: Deleting consensus metadata
W20260927 18:45:34.801781 6967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:45:34.802955 6953 catalog_manager.cc:4841] TS 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845): tablet 870cbc3e6a15412a9fde5a66217a4ed4 (table test-table [id=5566508dedeb4b08a6156b1b9122ba49]) successfully deleted
I20260927 18:45:34.803231 7548 ts_tablet_manager.cc:1918] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:34.803400 7548 log.cc:1196] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/870cbc3e6a15412a9fde5a66217a4ed4
I20260927 18:45:34.803867 7548 ts_tablet_manager.cc:1939] T 870cbc3e6a15412a9fde5a66217a4ed4 P add65bcb91c64ea7951a8f8a2edd9763: Deleting consensus metadata
I20260927 18:45:34.804744 7549 ts_tablet_manager.cc:1918] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:45:34.804955 6951 catalog_manager.cc:4841] TS add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057): tablet 870cbc3e6a15412a9fde5a66217a4ed4 (table test-table [id=5566508dedeb4b08a6156b1b9122ba49]) successfully deleted
I20260927 18:45:34.808799 7549 log.cc:1196] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/870cbc3e6a15412a9fde5a66217a4ed4
I20260927 18:45:34.809783 7549 ts_tablet_manager.cc:1939] T 870cbc3e6a15412a9fde5a66217a4ed4 P 72f541860ae04f3694eed63a55df0ebf: Deleting consensus metadata
I20260927 18:45:34.809852 7381 tablet_service.cc:1467] Processing CreateTablet for tablet cd411e6e776a4f75bb0dc046faa3d0fa (DEFAULT_TABLE table=test-table [id=8726f5c1643e4ed69a1983b55a2761f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:34.810299 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd411e6e776a4f75bb0dc046faa3d0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.813910 7243 tablet_service.cc:1467] Processing CreateTablet for tablet cd411e6e776a4f75bb0dc046faa3d0fa (DEFAULT_TABLE table=test-table [id=8726f5c1643e4ed69a1983b55a2761f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:34.813999 7104 tablet_service.cc:1467] Processing CreateTablet for tablet cd411e6e776a4f75bb0dc046faa3d0fa (DEFAULT_TABLE table=test-table [id=8726f5c1643e4ed69a1983b55a2761f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:45:34.814263 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd411e6e776a4f75bb0dc046faa3d0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.814445 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd411e6e776a4f75bb0dc046faa3d0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:34.817310 6951 catalog_manager.cc:4841] TS 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843): tablet 870cbc3e6a15412a9fde5a66217a4ed4 (table test-table [id=5566508dedeb4b08a6156b1b9122ba49]) successfully deleted
I20260927 18:45:34.819917 7479 tablet_bootstrap.cc:492] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Bootstrap starting.
I20260927 18:45:34.821610 7479 tablet_bootstrap.cc:654] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.822747 7478 tablet_bootstrap.cc:492] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Bootstrap starting.
I20260927 18:45:34.822755 7477 tablet_bootstrap.cc:492] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Bootstrap starting.
I20260927 18:45:34.824076 7478 tablet_bootstrap.cc:654] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.824331 7477 tablet_bootstrap.cc:654] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:34.832060 7478 tablet_bootstrap.cc:492] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: No bootstrap required, opened a new log
I20260927 18:45:34.832357 7478 ts_tablet_manager.cc:1397] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:45:34.833361 7478 raft_consensus.cc:348] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.833835 7478 raft_consensus.cc:374] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.833992 7478 raft_consensus.cc:729] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.834416 7478 consensus_queue.cc:260] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.839397 7478 ts_tablet_manager.cc:1428] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 18:45:34.843518 7477 tablet_bootstrap.cc:492] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: No bootstrap required, opened a new log
I20260927 18:45:34.843750 7477 ts_tablet_manager.cc:1397] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.007s
I20260927 18:45:34.844635 7477 raft_consensus.cc:348] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.844920 7477 raft_consensus.cc:374] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.845000 7477 raft_consensus.cc:729] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add65bcb91c64ea7951a8f8a2edd9763, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.845335 7477 consensus_queue.cc:260] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.846609 7479 tablet_bootstrap.cc:492] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: No bootstrap required, opened a new log
I20260927 18:45:34.846841 7479 ts_tablet_manager.cc:1397] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.008s
I20260927 18:45:34.847882 7479 raft_consensus.cc:348] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.848209 7479 raft_consensus.cc:374] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:34.848343 7479 raft_consensus.cc:729] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11602e61659546a78476b381cf8ef02e, State: Initialized, Role: FOLLOWER
I20260927 18:45:34.848393 7477 ts_tablet_manager.cc:1428] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:45:34.848739 7479 consensus_queue.cc:260] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:34.851467 7479 ts_tablet_manager.cc:1428] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 18:45:35.208585 7543 raft_consensus.cc:482] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:45:35.208850 7543 raft_consensus.cc:504] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:35.209592 7543 leader_election.cc:288] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:35.210124 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd411e6e776a4f75bb0dc046faa3d0fa" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11602e61659546a78476b381cf8ef02e" is_pre_election: true
I20260927 18:45:35.210206 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd411e6e776a4f75bb0dc046faa3d0fa" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763" is_pre_election: true
I20260927 18:45:35.210390 7268 raft_consensus.cc:2454] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 0.
I20260927 18:45:35.210516 7406 raft_consensus.cc:2454] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 0.
I20260927 18:45:35.210960 7056 leader_election.cc:302] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, 72f541860ae04f3694eed63a55df0ebf; no voters:
I20260927 18:45:35.211265 7543 raft_consensus.cc:2781] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:45:35.211387 7543 raft_consensus.cc:482] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:45:35.211469 7543 raft_consensus.cc:3037] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:35.214529 7543 raft_consensus.cc:504] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:35.215194 7543 leader_election.cc:288] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 election: Requested vote from peers 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845), add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057)
I20260927 18:45:35.216070 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd411e6e776a4f75bb0dc046faa3d0fa" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add65bcb91c64ea7951a8f8a2edd9763"
I20260927 18:45:35.216352 7406 raft_consensus.cc:3037] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:35.216400 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd411e6e776a4f75bb0dc046faa3d0fa" candidate_uuid: "72f541860ae04f3694eed63a55df0ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11602e61659546a78476b381cf8ef02e"
I20260927 18:45:35.216563 7268 raft_consensus.cc:3037] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:35.218798 7268 raft_consensus.cc:2454] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 1.
I20260927 18:45:35.219249 7056 leader_election.cc:302] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11602e61659546a78476b381cf8ef02e, 72f541860ae04f3694eed63a55df0ebf; no voters:
I20260927 18:45:35.219486 7406 raft_consensus.cc:2454] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f541860ae04f3694eed63a55df0ebf in term 1.
I20260927 18:45:35.219534 7543 raft_consensus.cc:2781] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:35.220104 7543 raft_consensus.cc:686] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 1 LEADER]: Becoming Leader. State: Replica: 72f541860ae04f3694eed63a55df0ebf, State: Running, Role: LEADER
I20260927 18:45:35.220430 7543 consensus_queue.cc:237] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } }
I20260927 18:45:35.226814 6967 catalog_manager.cc:5495] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 72f541860ae04f3694eed63a55df0ebf (127.31.26.193). New cstate: current_term: 1 leader_uuid: "72f541860ae04f3694eed63a55df0ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f541860ae04f3694eed63a55df0ebf" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260927 18:45:35.292390 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:35.287146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.31.26.193:42843)
0927 18:45:35.287707 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:35.287712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:35.287729 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:45:35.287812 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:35.287816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:35.287885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:35.287976 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:45:35.288362 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.289077 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.290514 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.291002 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.291109 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:35.291538 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:45:35.291716 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:45:35.291816 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:35.292011 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 18:45:35.292082 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":306,"threads_started":1}
I20260927 18:45:35.301802 7268 raft_consensus.cc:1261] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Refusing update from remote peer 72f541860ae04f3694eed63a55df0ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:35.301910 7406 raft_consensus.cc:1261] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Refusing update from remote peer 72f541860ae04f3694eed63a55df0ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:45:35.302443 7543 consensus_queue.cc:1035] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "add65bcb91c64ea7951a8f8a2edd9763" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:35.302712 7579 consensus_queue.cc:1035] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "11602e61659546a78476b381cf8ef02e" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:45:35.351460 6967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:35.351737 6967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:45:35.354445 6967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e00576785742d98da3518458c87156: Sending DeleteTablet for 3 replicas of tablet cd411e6e776a4f75bb0dc046faa3d0fa
I20260927 18:45:35.355361 7243 tablet_service.cc:1514] Processing DeleteTablet for tablet cd411e6e776a4f75bb0dc046faa3d0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:35 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:32990
I20260927 18:45:35.355361 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet cd411e6e776a4f75bb0dc046faa3d0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:35 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:39698
I20260927 18:45:35.355408 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet cd411e6e776a4f75bb0dc046faa3d0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:45:35 UTC) from {username='oryx', principal='oryx/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:59892
I20260927 18:45:35.356628 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 7038
I20260927 18:45:35.356678 7594 tablet_replica.cc:330] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: stopping tablet replica
I20260927 18:45:35.357115 7595 tablet_replica.cc:330] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: stopping tablet replica
I20260927 18:45:35.357321 7595 raft_consensus.cc:2229] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:35.357584 7595 raft_consensus.cc:2258] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:35.357743 7596 tablet_replica.cc:330] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: stopping tablet replica
I20260927 18:45:35.357936 7594 raft_consensus.cc:2229] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:45:35.358060 7596 raft_consensus.cc:2229] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:45:35.358264 7596 raft_consensus.cc:2258] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:35.358300 7594 raft_consensus.cc:2258] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:45:35.358875 7595 ts_tablet_manager.cc:1905] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:35.359710 7596 ts_tablet_manager.cc:1905] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:35.361742 7594 ts_tablet_manager.cc:1905] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:45:35.365118 7595 ts_tablet_manager.cc:1918] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:45:35.365270 7595 log.cc:1196] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/cd411e6e776a4f75bb0dc046faa3d0fa
I20260927 18:45:35.365775 7595 ts_tablet_manager.cc:1939] T cd411e6e776a4f75bb0dc046faa3d0fa P 11602e61659546a78476b381cf8ef02e: Deleting consensus metadata
I20260927 18:45:35.367076 6953 catalog_manager.cc:4841] TS 11602e61659546a78476b381cf8ef02e (127.31.26.194:36845): tablet cd411e6e776a4f75bb0dc046faa3d0fa (table test-table [id=8726f5c1643e4ed69a1983b55a2761f7]) successfully deleted
I20260927 18:45:35.368731 7594 ts_tablet_manager.cc:1918] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:45:35.368901 7594 log.cc:1196] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/cd411e6e776a4f75bb0dc046faa3d0fa
I20260927 18:45:35.369314 7596 ts_tablet_manager.cc:1918] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:45:35.369451 7596 log.cc:1196] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/cd411e6e776a4f75bb0dc046faa3d0fa
I20260927 18:45:35.369469 7594 ts_tablet_manager.cc:1939] T cd411e6e776a4f75bb0dc046faa3d0fa P 72f541860ae04f3694eed63a55df0ebf: Deleting consensus metadata
I20260927 18:45:35.369850 7596 ts_tablet_manager.cc:1939] T cd411e6e776a4f75bb0dc046faa3d0fa P add65bcb91c64ea7951a8f8a2edd9763: Deleting consensus metadata
I20260927 18:45:35.370584 6951 catalog_manager.cc:4841] TS 72f541860ae04f3694eed63a55df0ebf (127.31.26.193:42843): tablet cd411e6e776a4f75bb0dc046faa3d0fa (table test-table [id=8726f5c1643e4ed69a1983b55a2761f7]) successfully deleted
I20260927 18:45:35.370878 6951 catalog_manager.cc:4841] TS add65bcb91c64ea7951a8f8a2edd9763 (127.31.26.195:40057): tablet cd411e6e776a4f75bb0dc046faa3d0fa (table test-table [id=8726f5c1643e4ed69a1983b55a2761f7]) successfully deleted
Sep 27 18:45:35 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.193@KRBTEST.COM
I20260927 18:45:35.384227 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:35.358846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.31.26.193:42843)
0927 18:45:35.359019 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:35.359024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:35.369855 (+ 10831us) server_negotiation.cc:407] Connection header received
0927 18:45:35.370034 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:35.370043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:35.370129 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:35.370250 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:35.371749 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.372438 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.373821 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.374333 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.374455 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:35.378462 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:35.378478 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:35.378494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:35.378534 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:35.381563 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:35.382256 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:35.382265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:35.382267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:35.382419 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:35.382782 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:35.382787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:35.382790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:35.383137 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:35.383368 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:35.383649 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 18:45:35.383876 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:45:35.384866 7169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:35.523566 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 7038
I20260927 18:45:35.547262 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 7178
Sep 27 18:45:35 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.194@KRBTEST.COM
I20260927 18:45:35.563089 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:35.548107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.31.26.194:36845)
0927 18:45:35.548576 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:35.548580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:35.548604 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:45:35.548660 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:35.548664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:35.548729 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:35.548867 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:35.550219 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.550987 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.552242 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.552649 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.552750 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:35.557288 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:35.557304 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:35.557321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:35.557361 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:35.560365 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:35.561128 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:35.561142 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:35.561148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:35.561340 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:35.561766 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:35.561769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:35.561772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:35.562099 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:35.562345 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:35.562646 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 18:45:35.562818 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260927 18:45:35.563889 7308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:35.706496 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 7178
I20260927 18:45:35.731472 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 7317
Sep 27 18:45:35 dist-test-slave-rfd5 krb5kdc[6922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.26.195@KRBTEST.COM
I20260927 18:45:35.748113 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:35.732248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.31.26.195:40057)
0927 18:45:35.732782 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:35.732786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:35.732802 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:45:35.732855 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:35.732859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:35.732918 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:35.733121 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:35.734655 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.735415 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.736655 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:35.737065 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:35.737243 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:35.742229 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:35.742246 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:35.742263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:35.742300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:35.745270 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:35.746012 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:35.746020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:35.746022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:35.746157 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:35.746535 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:35.746540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:35.746544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:35.746915 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:35.747218 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:35.747540 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:45:35.747770 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":320,"threads_started":1}
I20260927 18:45:35.748764 7447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:35.870042 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 7317
I20260927 18:45:35.891661 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 6935
I20260927 18:45:35.892865 7007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:36.008279 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 6935
2026-09-27T18:45:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7410 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:36 dist-test-slave-rfd5 krb5kdc[7606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:36 dist-test-slave-rfd5 krb5kdc[7606](info): set up 2 sockets
Sep 27 18:45:36 dist-test-slave-rfd5 krb5kdc[7606](info): commencing operation
krb5kdc: starting...
W20260927 18:45:37.110322 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:37 dist-test-slave-rfd5 krb5kdc[7606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:45:37.336539 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:38327
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:37.672996 7619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:37.673369 7619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:37.673550 7619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:37.681108 7619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:37.681232 7619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:37.681337 7619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:37.681440 7619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:37.681532 7619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:37.691761 7619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:38327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:37.693193 7619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:37.694574 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:37.707408 7624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:37.707490 7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:37.707468 7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:37.708062 7619 server_base.cc:1034] running on GCE node
I20260927 18:45:37.708905 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:37.710543 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:37.711782 7619 hybrid_clock.cc:648] HybridClock initialized: now 1790534737711737 us; error 45 us; skew 500 ppm
I20260927 18:45:37.712167 7619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T18:45:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1775 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:45:37 dist-test-slave-rfd5 krb5kdc[7633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:45:37 dist-test-slave-rfd5 krb5kdc[7633](info): set up 2 sockets
Sep 27 18:45:37 dist-test-slave-rfd5 krb5kdc[7633](info): commencing operation
krb5kdc: starting...
W20260927 18:45:38.906138 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:45:39 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:45:39.136502 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34619
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:34619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:39.471390 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:39.471721 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:39.471812 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:39.479327 7646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:39.479463 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:39.479511 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:39.479565 7646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:39.479609 7646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:39.489886 7646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:34619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:39.491250 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:39.492560 7646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:39.505692 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:39.505770 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:39.505770 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:39.506125 7646 server_base.cc:1034] running on GCE node
I20260927 18:45:39.507061 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:39.508574 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:39.509881 7646 hybrid_clock.cc:648] HybridClock initialized: now 1790534739509786 us; error 85 us; skew 500 ppm
I20260927 18:45:39.510293 7646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:39 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:39.517961 7646 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:45:39.519681 7646 webserver.cc:460] Webserver started at http://127.31.26.254:33097/ using document root <none> and password file <none>
I20260927 18:45:39.520447 7646 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:39.520627 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:39.520974 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:39.524243 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "9b8a8faf8c3c46be8d5af543168cc21b"
format_stamp: "Formatted at 2026-09-27 18:45:39 on dist-test-slave-rfd5"
server_key: "c1e1ccf9771cc413193142b25a68c743"
server_key_iv: "78701b2531c501083b5b652be5484045"
server_key_version: "encryptionkey@0"
I20260927 18:45:39.525169 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "9b8a8faf8c3c46be8d5af543168cc21b"
format_stamp: "Formatted at 2026-09-27 18:45:39 on dist-test-slave-rfd5"
server_key: "c1e1ccf9771cc413193142b25a68c743"
server_key_iv: "78701b2531c501083b5b652be5484045"
server_key_version: "encryptionkey@0"
I20260927 18:45:39.530818 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:45:39.534603 7661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:39.536154 7646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:45:39.536373 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "9b8a8faf8c3c46be8d5af543168cc21b"
format_stamp: "Formatted at 2026-09-27 18:45:39 on dist-test-slave-rfd5"
server_key: "c1e1ccf9771cc413193142b25a68c743"
server_key_iv: "78701b2531c501083b5b652be5484045"
server_key_version: "encryptionkey@0"
I20260927 18:45:39.536607 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:39.561228 7646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:39.565840 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:39.566139 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:39.585847 7646 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:34619
I20260927 18:45:39.585877 7712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:34619 every 8 connection(s)
I20260927 18:45:39.587234 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:39.588533 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 7646
I20260927 18:45:39.588768 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:45:39.589593 31851 external_mini_cluster.cc:1428] Setting key ebcbe6d35d36ee39331b68987042ed69
I20260927 18:45:39.592264 7713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:45:39.603399 7713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Bootstrap starting.
Sep 27 18:45:39 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:39.607143 7713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Neither blocks nor log segments found. Creating new log.
I20260927 18:45:39.608269 7713 log.cc:824] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:39.611945 7713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: No bootstrap required, opened a new log
I20260927 18:45:39.612924 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:39.591781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.31.26.254:34619)
0927 18:45:39.592602 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:39.592614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:39.592656 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:45:39.594378 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:39.594401 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:39.594815 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:39.595268 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:39.596814 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:39.598235 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:39.599625 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:39.600370 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:39.600584 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:39.605006 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:39.605022 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:39.605039 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:39.605139 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:39.608461 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:39.609413 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:39.609429 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:39.609441 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:39.609673 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:39.610178 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:39.610182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:39.610184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:39.610722 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:39.610955 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:39.611452 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 18:45:39.611800 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":357,"threads_started":1}
I20260927 18:45:39.619705 7713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:39.620121 7713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:45:39.620290 7713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8a8faf8c3c46be8d5af543168cc21b, State: Initialized, Role: FOLLOWER
I20260927 18:45:39.621014 7713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:39.621359 7713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:39.621506 7713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:39.621706 7713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:45:39.623967 7713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:39.624559 7713 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8a8faf8c3c46be8d5af543168cc21b; no voters:
I20260927 18:45:39.625345 7713 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:45:39.625504 7718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:45:39.626307 7718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 LEADER]: Becoming Leader. State: Replica: 9b8a8faf8c3c46be8d5af543168cc21b, State: Running, Role: LEADER
I20260927 18:45:39.626886 7718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:39.627715 7713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:39.630606 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } } }
I20260927 18:45:39.631031 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:39.631438 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8a8faf8c3c46be8d5af543168cc21b. Latest consensus state: current_term: 1 leader_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } } }
I20260927 18:45:39.631821 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:39.637868 7726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:39.643181 7726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:39.652071 7726 catalog_manager.cc:1347] Generated new cluster ID: dc7bb5de3b064686839f432d17c2e6f3
I20260927 18:45:39.652202 7726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:39.669186 7726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:45:39.670694 7726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:39.684150 7726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Generated new TSK 0
I20260927 18:45:39.685386 7726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:45:39.752597 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34619
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:40.092411 7739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:40.092720 7739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:40.092813 7739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:40.101218 7739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:40.101326 7739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:40.101441 7739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:40.112953 7739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:40.114905 7739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:40.116846 7739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:40.131475 7747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:40.131415 7745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:40.131454 7744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:40.132432 7739 server_base.cc:1034] running on GCE node
I20260927 18:45:40.133276 7739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:40.134302 7739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:40.135542 7739 hybrid_clock.cc:648] HybridClock initialized: now 1790534740135464 us; error 62 us; skew 500 ppm
I20260927 18:45:40.135918 7739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:40 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:40.139271 7739 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:45:40.140955 7739 webserver.cc:460] Webserver started at http://127.31.26.193:32871/ using document root <none> and password file <none>
I20260927 18:45:40.141826 7739 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:40.141978 7739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:40.142329 7739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:40.145604 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "bbf77eb266424867a7e1ea55e582d9d2"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "7b302254bea8cec20e543a8c5e3c5bd5"
server_key_iv: "a20ba4f3f6b689c796f4b87a09bba5b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:40.146448 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "bbf77eb266424867a7e1ea55e582d9d2"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "7b302254bea8cec20e543a8c5e3c5bd5"
server_key_iv: "a20ba4f3f6b689c796f4b87a09bba5b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:40.151932 7739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:45:40.155823 7754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:40.157473 7739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:45:40.157666 7739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "bbf77eb266424867a7e1ea55e582d9d2"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "7b302254bea8cec20e543a8c5e3c5bd5"
server_key_iv: "a20ba4f3f6b689c796f4b87a09bba5b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:40.157855 7739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:40.185387 7739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:40.190153 7739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:40.190472 7739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:40.191452 7739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:40.192829 7739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:40.192905 7739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:40.193033 7739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:40.193130 7739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:40.226197 7739 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:41981
I20260927 18:45:40.226250 7866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:41981 every 8 connection(s)
I20260927 18:45:40.227582 7739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:40.236467 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 7739
I20260927 18:45:40.236696 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:45:40.237385 31851 external_mini_cluster.cc:1428] Setting key 511a087e9482e4e8247e10a6741671ff
Sep 27 18:45:40 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:40.250664 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:40.230441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:59943 (local address 127.31.26.254:34619)
0927 18:45:40.230889 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:40.230895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:40.232035 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 18:45:40.234427 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:40.234433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:40.234533 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:40.234656 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:40.236702 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.237627 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.239594 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.240080 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.240216 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:40.244003 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:40.244013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:40.244021 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:40.244066 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:40.247317 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:40.248273 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:40.248277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:40.248279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:40.248397 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:40.248888 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:40.248891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:40.248893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:40.249223 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:40.249334 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:40.250239 (+ 905us) server_negotiation.cc:299] Negotiation successful
0927 18:45:40.250408 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
I20260927 18:45:40.252306 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:40.231121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:59943)
0927 18:45:40.231881 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:40.231924 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:40.234124 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:40.234908 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:40.234917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:40.235633 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:40.236487 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:40.236514 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.237794 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.237799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:40.239450 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:40.239471 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.240314 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.240322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:40.240599 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:40.241411 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:40.241434 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:40.243770 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:40.247475 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:40.247485 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:40.247506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:40.248156 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:40.248510 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:40.248517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:40.248523 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:40.248781 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:40.249346 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:40.249357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:40.249815 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 18:45:40.250139 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 18:45:40.250736 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":266,"threads_started":1}
I20260927 18:45:40.254071 7867 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34619
I20260927 18:45:40.254400 7867 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:40.255155 7867 heartbeater.cc:511] Master 127.31.26.254:34619 requested a full tablet report, sending...
I20260927 18:45:40.257550 7678 ts_manager.cc:194] Registered new tserver with Master: bbf77eb266424867a7e1ea55e582d9d2 (127.31.26.193:41981)
I20260927 18:45:40.259039 7678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:59943
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:45:40.289355 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34619
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:40.623549 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:40.623862 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:40.624022 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:40.631712 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:40.631863 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:40.632009 7873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:40.642346 7873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:40.643656 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:40.644928 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:40.659051 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:40.659137 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:40.659088 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:40.665446 7873 server_base.cc:1034] running on GCE node
I20260927 18:45:40.666235 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:40.667392 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:40.668651 7873 hybrid_clock.cc:648] HybridClock initialized: now 1790534740668592 us; error 42 us; skew 500 ppm
I20260927 18:45:40.669073 7873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:40 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:40.672647 7873 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:45:40.674413 7873 webserver.cc:460] Webserver started at http://127.31.26.194:34193/ using document root <none> and password file <none>
I20260927 18:45:40.675186 7873 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:40.675320 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:40.675653 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:40.678705 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "067af83fcab74fb593bfa77475c5cca0"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "65ee2663643c9c5bdb49d487d4312521"
server_key_iv: "7bc447466b24d7e13b0c07a5d1b875e1"
server_key_version: "encryptionkey@0"
I20260927 18:45:40.679507 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "067af83fcab74fb593bfa77475c5cca0"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "65ee2663643c9c5bdb49d487d4312521"
server_key_iv: "7bc447466b24d7e13b0c07a5d1b875e1"
server_key_version: "encryptionkey@0"
I20260927 18:45:40.685031 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:45:40.688524 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:40.690038 7873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:40.690235 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "067af83fcab74fb593bfa77475c5cca0"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "65ee2663643c9c5bdb49d487d4312521"
server_key_iv: "7bc447466b24d7e13b0c07a5d1b875e1"
server_key_version: "encryptionkey@0"
I20260927 18:45:40.690447 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:40.716936 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:40.721784 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:40.722112 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:40.723196 7873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:40.724728 7873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:40.724805 7873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:40.724943 7873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:40.724993 7873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:40.760823 7873 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:35471
I20260927 18:45:40.760942 8000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:35471 every 8 connection(s)
I20260927 18:45:40.762352 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:40.772439 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 7873
I20260927 18:45:40.772652 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:45:40.773315 31851 external_mini_cluster.cc:1428] Setting key 4fc40c494e16b671f163feadfe1b0f0b
Sep 27 18:45:40 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:40.784209 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:40.765174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:35837 (local address 127.31.26.254:34619)
0927 18:45:40.765624 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:40.765629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:40.766603 (+ 974us) server_negotiation.cc:407] Connection header received
0927 18:45:40.768483 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:40.768488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:40.768595 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:40.768715 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:40.770381 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.771087 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.772834 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.773525 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.773707 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:40.777751 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:40.777756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:40.777759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:40.777793 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:40.780418 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:40.781289 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:40.781294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:40.781296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:40.781421 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:40.782056 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:40.782062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:40.782066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:40.782395 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:40.782513 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:40.783423 (+ 910us) server_negotiation.cc:299] Negotiation successful
0927 18:45:40.783678 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":241,"threads_started":1}
I20260927 18:45:40.784837 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:40.765643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:35837)
0927 18:45:40.766416 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:40.766454 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:40.768183 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:40.768970 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:40.768978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:40.769611 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:40.770195 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:40.770213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.771235 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.771240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:40.772646 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:40.772663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:40.773779 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:40.773785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:40.773994 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:40.774815 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:40.774839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:40.777483 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:40.780582 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:40.780592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:40.780612 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:40.781139 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:40.781593 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:40.781599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:40.781604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:40.781888 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:40.782541 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:40.782553 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:40.783011 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 18:45:40.783260 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 18:45:40.783681 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":282,"threads_started":1}
I20260927 18:45:40.786419 8001 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34619
I20260927 18:45:40.786752 8001 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:40.787470 8001 heartbeater.cc:511] Master 127.31.26.254:34619 requested a full tablet report, sending...
I20260927 18:45:40.789022 7678 ts_manager.cc:194] Registered new tserver with Master: 067af83fcab74fb593bfa77475c5cca0 (127.31.26.194:35471)
I20260927 18:45:40.789878 7678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:35837
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:45:40.824544 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34619
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:41.177453 8007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:41.177831 8007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:41.178021 8007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:41.185627 8007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:41.185834 8007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:41.186004 8007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:41.196568 8007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:41.197953 8007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:41.199424 8007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:41.213802 8015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:41.213852 8012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:41.213815 8013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:41.214771 8007 server_base.cc:1034] running on GCE node
I20260927 18:45:41.215425 8007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:41.216435 8007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:41.217661 8007 hybrid_clock.cc:648] HybridClock initialized: now 1790534741217616 us; error 38 us; skew 500 ppm
I20260927 18:45:41.218036 8007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:41 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534741, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:41.221372 8007 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:45:41.222858 8007 webserver.cc:460] Webserver started at http://127.31.26.195:39127/ using document root <none> and password file <none>
I20260927 18:45:41.223614 8007 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:41.223774 8007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:41.224092 8007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:45:41.227340 8007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "52955f40780d4d3292b5e38fa1532838"
format_stamp: "Formatted at 2026-09-27 18:45:41 on dist-test-slave-rfd5"
server_key: "5d10037f00ebebd9318f195de92b4dd9"
server_key_iv: "eb99993c3454ce1a72b366a05512b891"
server_key_version: "encryptionkey@0"
I20260927 18:45:41.228250 8007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "52955f40780d4d3292b5e38fa1532838"
format_stamp: "Formatted at 2026-09-27 18:45:41 on dist-test-slave-rfd5"
server_key: "5d10037f00ebebd9318f195de92b4dd9"
server_key_iv: "eb99993c3454ce1a72b366a05512b891"
server_key_version: "encryptionkey@0"
I20260927 18:45:41.234194 8007 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:45:41.238431 8022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:41.240383 8007 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:45:41.240576 8007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "52955f40780d4d3292b5e38fa1532838"
format_stamp: "Formatted at 2026-09-27 18:45:41 on dist-test-slave-rfd5"
server_key: "5d10037f00ebebd9318f195de92b4dd9"
server_key_iv: "eb99993c3454ce1a72b366a05512b891"
server_key_version: "encryptionkey@0"
I20260927 18:45:41.240808 8007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:41.262748 7867 heartbeater.cc:503] Master 127.31.26.254:34619 was elected leader, sending a full tablet report...
I20260927 18:45:41.280702 8007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:41.285598 8007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:41.285910 8007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:41.286893 8007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:41.288285 8007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:41.288386 8007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:41.288549 8007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:41.288622 8007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:41.322675 8007 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:37073
I20260927 18:45:41.322726 8134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:37073 every 8 connection(s)
I20260927 18:45:41.323997 8007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:41.328769 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8007
I20260927 18:45:41.328944 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:45:41.329566 31851 external_mini_cluster.cc:1428] Setting key 773a29552ac1c1f31ba53377c30167f3
Sep 27 18:45:41 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534741, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:45:41.345420 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:41.326727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:50977 (local address 127.31.26.254:34619)
0927 18:45:41.327198 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:41.327204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:41.328243 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 18:45:41.330416 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:41.330420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:41.330517 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:41.330642 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:41.332478 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.333513 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.335078 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.335470 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.335568 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:41.338956 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:41.338962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:41.338965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:41.338998 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:41.341840 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.342640 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.342644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.342646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.342776 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.343222 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.343229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.343232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.343577 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:41.343745 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:41.344745 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0927 18:45:41.344970 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260927 18:45:41.346607 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:41.327380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:50977)
0927 18:45:41.328069 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:41.328107 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:41.330098 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:41.330924 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:41.330934 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:41.331578 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:41.332276 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:41.332293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.333649 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.333653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:41.334932 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:41.334943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.335659 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.335664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:41.335862 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:41.336539 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:41.336560 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:41.338759 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:45:41.342020 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:41.342027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:41.342043 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:41.342531 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:41.342872 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:45:41.342875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:45:41.342877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:45:41.343073 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:45:41.343764 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:45:41.343773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:45:41.344255 (+ 482us) client_negotiation.cc:769] Sending connection context
0927 18:45:41.344625 (+ 370us) client_negotiation.cc:240] Negotiation successful
0927 18:45:41.345247 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":278,"threads_started":1}
I20260927 18:45:41.348030 8135 heartbeater.cc:347] Connected to a master server at 127.31.26.254:34619
I20260927 18:45:41.348361 8135 heartbeater.cc:464] Registering TS with master...
I20260927 18:45:41.349139 8135 heartbeater.cc:511] Master 127.31.26.254:34619 requested a full tablet report, sending...
I20260927 18:45:41.350580 7678 ts_manager.cc:194] Registered new tserver with Master: 52955f40780d4d3292b5e38fa1532838 (127.31.26.195:37073)
I20260927 18:45:41.351399 7678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:50977
I20260927 18:45:41.357774 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab.
Entry for principal oryx/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab.
WARNING: no policy specified for oryx/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab.
Entry for principal oryx/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab.
WARNING: no policy specified for oryx/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab.
Entry for principal oryx/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab.
I20260927 18:45:41.505520 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 7739
Sep 27 18:45:41 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:45:41.523557 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:41.506810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.31.26.193:41981)
0927 18:45:41.507475 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:41.507483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:41.507511 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:45:41.507573 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:41.507587 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:41.507814 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:41.507960 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:41.509602 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.510720 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.512091 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.512728 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.512847 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:41.517536 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:41.517553 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:41.517571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:41.517622 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:41.520462 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.521282 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.521290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.521293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.521434 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.521848 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.521852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.521855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.522306 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:41.522545 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:41.522978 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 18:45:41.523238 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":341,"threads_started":1}
I20260927 18:45:41.524976 7862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:41.609629 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 7739
I20260927 18:45:41.626060 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 7873
Sep 27 18:45:41 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:45:41.642776 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:41.627068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.31.26.194:35471)
0927 18:45:41.627540 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:41.627547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:41.627571 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:45:41.627628 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:41.627634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:41.627866 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:41.628122 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:41.629392 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.630460 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.631744 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.632353 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.632455 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:41.636977 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:41.636992 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:41.637009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:41.637057 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:41.639892 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.640665 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.640673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.640676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.640818 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.641291 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.641294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.641296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.641641 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:41.641890 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:41.642172 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 18:45:41.642374 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":236,"threads_started":1}
I20260927 18:45:41.644155 7996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:41.726924 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 7873
I20260927 18:45:41.743340 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8007
Sep 27 18:45:41 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:45:41.759814 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:41.744371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.31.26.195:37073)
0927 18:45:41.744887 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:41.744895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:41.744925 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:45:41.744987 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:41.744993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:41.745285 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:41.745457 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:41.746766 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.747757 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.749038 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:41.749738 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:41.749841 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:41.754168 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:41.754181 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:41.754197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:41.754238 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:41.757233 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.757903 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.757910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.757912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.758033 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:41.758366 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:41.758369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:41.758371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:41.758717 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:41.758925 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:41.759199 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 18:45:41.759396 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260927 18:45:41.761261 8130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:41.844352 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8007
I20260927 18:45:41.860260 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 7646
I20260927 18:45:41.861519 7708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:41.962641 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 7646
I20260927 18:45:41.983533 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34619
--webserver_interface=127.31.26.254
--webserver_port=33097
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:34619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:45:42.335764 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:42.336086 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:42.336249 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:42.343787 8153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:45:42.343919 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:42.344007 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:42.344117 8153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:45:42.344192 8153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:45:42.354353 8153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:34619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=33097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:42.355773 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:42.357162 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:42.368996 8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:42.368978 8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:42.369230 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:42.370442 8153 server_base.cc:1034] running on GCE node
I20260927 18:45:42.371382 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:42.372906 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:42.374145 8153 hybrid_clock.cc:648] HybridClock initialized: now 1790534742374094 us; error 63 us; skew 500 ppm
I20260927 18:45:42.374506 8153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:42 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534742, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:42.377744 8153 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:45:42.379343 8153 webserver.cc:460] Webserver started at http://127.31.26.254:33097/ using document root <none> and password file <none>
I20260927 18:45:42.380062 8153 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:42.380230 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:42.385695 8153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:45:42.388509 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:42.389935 8153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:45:42.390105 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "9b8a8faf8c3c46be8d5af543168cc21b"
format_stamp: "Formatted at 2026-09-27 18:45:39 on dist-test-slave-rfd5"
server_key: "c1e1ccf9771cc413193142b25a68c743"
server_key_iv: "78701b2531c501083b5b652be5484045"
server_key_version: "encryptionkey@0"
I20260927 18:45:42.391057 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:42.414239 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:42.418476 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:42.418696 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:42.437819 8153 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:34619
I20260927 18:45:42.437844 8219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:34619 every 8 connection(s)
I20260927 18:45:42.439431 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:45:42.446472 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8153
I20260927 18:45:42.446868 8220 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:45:42.447424 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:41981
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:34619
--webserver_port=32871
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:45:42.450455 8220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Bootstrap starting.
I20260927 18:45:42.469836 8220 log.cc:824] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:45:42.479236 8220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:45:42.479815 8220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Bootstrap complete.
I20260927 18:45:42.489024 8220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:42.489984 8220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8a8faf8c3c46be8d5af543168cc21b, State: Initialized, Role: FOLLOWER
I20260927 18:45:42.490772 8220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:42.491020 8220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:45:42.491137 8220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:45:42.491331 8220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:45:42.495846 8220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:42.496423 8220 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8a8faf8c3c46be8d5af543168cc21b; no voters:
I20260927 18:45:42.497118 8220 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:45:42.497272 8224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:45:42.498729 8224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [term 2 LEADER]: Becoming Leader. State: Replica: 9b8a8faf8c3c46be8d5af543168cc21b, State: Running, Role: LEADER
I20260927 18:45:42.499408 8224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } }
I20260927 18:45:42.499683 8220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:45:42.501901 8226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8a8faf8c3c46be8d5af543168cc21b. Latest consensus state: current_term: 2 leader_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } } }
I20260927 18:45:42.502301 8226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:42.502871 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8a8faf8c3c46be8d5af543168cc21b" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34619 } } }
I20260927 18:45:42.503198 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b [sys.catalog]: This master's current role is: LEADER
I20260927 18:45:42.514339 8230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:45:42.524593 8230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:45:42.527395 8230 catalog_manager.cc:1259] Loaded cluster ID: dc7bb5de3b064686839f432d17c2e6f3
I20260927 18:45:42.527604 8230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:45:42.531888 8230 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:45:42.533629 8230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b8a8faf8c3c46be8d5af543168cc21b: Loaded TSK: 0
I20260927 18:45:42.535394 8230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:45:42.811697 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:42.812074 8222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:42.812240 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:42.819849 8222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:42.820017 8222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:42.820196 8222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:45:42.830699 8222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.193.keytab
--principal=oryx/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:41981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=32871
--tserver_master_addrs=127.31.26.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:42.832176 8222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:42.833617 8222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:42.849364 8246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:42.849319 8247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:42.850507 8249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:42.850847 8222 server_base.cc:1034] running on GCE node
I20260927 18:45:42.851572 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:42.852655 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:42.853874 8222 hybrid_clock.cc:648] HybridClock initialized: now 1790534742853803 us; error 69 us; skew 500 ppm
I20260927 18:45:42.854221 8222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:42 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534742, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:42.857523 8222 init.cc:377] Logged in from keytab as oryx/127.31.26.193@KRBTEST.COM (short username oryx)
I20260927 18:45:42.859264 8222 webserver.cc:460] Webserver started at http://127.31.26.193:32871/ using document root <none> and password file <none>
I20260927 18:45:42.860008 8222 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:42.860167 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:42.866034 8222 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:45:42.869122 8256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:42.870630 8222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:45:42.870842 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "bbf77eb266424867a7e1ea55e582d9d2"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "7b302254bea8cec20e543a8c5e3c5bd5"
server_key_iv: "a20ba4f3f6b689c796f4b87a09bba5b0"
server_key_version: "encryptionkey@0"
I20260927 18:45:42.871833 8222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:42.897877 8222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:42.903151 8222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:42.903461 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:42.904554 8222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:42.906102 8222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:42.906243 8222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:42.906386 8222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:42.906477 8222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:42.941287 8222 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:41981
I20260927 18:45:42.941344 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:41981 every 8 connection(s)
I20260927 18:45:42.942886 8222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:45:42.950815 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8222
I20260927 18:45:42.951445 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:35471
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:34619
--webserver_port=34193
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:45:42 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:42 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:45:43.296404 8373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:43.296751 8373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:43.296912 8373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:43.304519 8373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:43.304699 8373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:43.304883 8373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:45:43.314947 8373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.194.keytab
--principal=oryx/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:35471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=34193
--tserver_master_addrs=127.31.26.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:43.316422 8373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:43.317883 8373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:43.337234 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.337393 8379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:43.337988 8382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:43.338392 8373 server_base.cc:1034] running on GCE node
I20260927 18:45:43.339637 8373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:43.341471 8373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:43.342794 8373 hybrid_clock.cc:648] HybridClock initialized: now 1790534743342687 us; error 76 us; skew 500 ppm
I20260927 18:45:43.343362 8373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:43 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534743, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:45:43.349442 8373 init.cc:377] Logged in from keytab as oryx/127.31.26.194@KRBTEST.COM (short username oryx)
I20260927 18:45:43.352099 8373 webserver.cc:460] Webserver started at http://127.31.26.194:34193/ using document root <none> and password file <none>
I20260927 18:45:43.353615 8373 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:43.353883 8373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:43.364585 8373 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 18:45:43.369815 8389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.372395 8373 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:45:43.372726 8373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "067af83fcab74fb593bfa77475c5cca0"
format_stamp: "Formatted at 2026-09-27 18:45:40 on dist-test-slave-rfd5"
server_key: "65ee2663643c9c5bdb49d487d4312521"
server_key_iv: "7bc447466b24d7e13b0c07a5d1b875e1"
server_key_version: "encryptionkey@0"
I20260927 18:45:43.375054 8373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:43.409027 8373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:43.416833 8373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:43.417309 8373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:43.418991 8373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:43.421479 8373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:43.421667 8373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.421824 8373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:43.421937 8373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:43.469216 8373 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:35471
I20260927 18:45:43.469290 8501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:35471 every 8 connection(s)
I20260927 18:45:43.471908 8373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:45:43.478700 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8373
I20260927 18:45:43.479467 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:37073
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:34619
--webserver_port=39127
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:45:43 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:43 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:45:44.162060 8506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:45:44.162449 8506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:45:44.162663 8506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:45:44.174552 8506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:45:44.174708 8506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:45:44.174953 8506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:45:44.192051 8506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/krb5kdc/oryx_127.31.26.195.keytab
--principal=oryx/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:37073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=39127
--tserver_master_addrs=127.31.26.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:45:44.193857 8506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:45:44.195602 8506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:45:44.217100 8513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:44.217061 8512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:45:44.230809 8515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:45:44.231456 8506 server_base.cc:1034] running on GCE node
I20260927 18:45:44.232286 8506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:45:44.233697 8506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:45:44.235234 8506 hybrid_clock.cc:648] HybridClock initialized: now 1790534744235146 us; error 101 us; skew 500 ppm
I20260927 18:45:44.235648 8506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534744, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:45:44.239053 8372 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.282s user 0.000s sys 0.018s
I20260927 18:45:44.239499 8506 init.cc:377] Logged in from keytab as oryx/127.31.26.195@KRBTEST.COM (short username oryx)
I20260927 18:45:44.241891 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:42.946385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:38111)
0927 18:45:42.947216 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:42.947254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:42.948997 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:42.950622 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:42.950633 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:42.951308 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:42.951998 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:42.952020 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:42.953752 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:42.953762 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:42.955021 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:42.955035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:42.956151 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:42.956158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:42.956449 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:42.957262 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:42.957287 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.240266 (+1282979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":354,"threads_started":1}
W20260927 18:45:44.241938 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:42.946226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:38111 (local address 127.31.26.254:34619)
0927 18:45:42.947056 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:42.947074 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:42.947392 (+ 318us) server_negotiation.cc:407] Connection header received
0927 18:45:42.949682 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:42.949696 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:42.950053 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:42.950462 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:42.952211 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:42.953562 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:42.955184 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:42.955890 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:42.956119 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.240791 (+1284672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:38111: BlockingRecv error: recv got EOF from 127.31.26.193:38111 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":259,"threads_started":1}
W20260927 18:45:44.242976 8369 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:34619 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.26.254:34619: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
I20260927 18:45:44.245047 8506 webserver.cc:460] Webserver started at http://127.31.26.195:39127/ using document root <none> and password file <none>
I20260927 18:45:44.245997 8506 fs_manager.cc:359] Metadata directory not provided
I20260927 18:45:44.246172 8506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:45:44.255412 8506 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:44.258968 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.243769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:33021)
0927 18:45:44.244139 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.244159 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.244273 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.244969 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.244974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.245454 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.245663 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.245675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.249723 (+ 4048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.249729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.253341 (+ 3612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.253358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.254161 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.254168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.254355 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.254995 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.255015 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.258669 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247}
W20260927 18:45:44.259984 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:44.244232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:33021 (local address 127.31.26.254:34619)
0927 18:45:44.244484 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.244491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.244508 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:45:44.244569 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.244574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.244652 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.244766 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.245886 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.249434 (+ 3548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.253525 (+ 4091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.253931 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.254122 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.259721 (+ 5599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:33021: BlockingRecv error: recv got EOF from 127.31.26.193:33021 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:44.273775 8522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:44.274061 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.259729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:41005)
0927 18:45:44.260574 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.260596 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.260727 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.261378 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.261385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.261895 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.262161 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.262177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.263917 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.263925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.265891 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.265911 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.267343 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.267352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.267609 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.268420 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.268452 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.273753 (+ 5301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":13119,"threads_started":1}
W20260927 18:45:44.274444 8369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:45:44.274541 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:44.260198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:41005 (local address 127.31.26.254:34619)
0927 18:45:44.260372 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.260378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.260848 (+ 470us) server_negotiation.cc:407] Connection header received
0927 18:45:44.260917 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.260925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.261051 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.261171 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.262356 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.263714 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.266100 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.266946 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.267096 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.274140 (+ 7044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:41005: BlockingRecv error: recv got EOF from 127.31.26.193:41005 (error 108)
Metrics: {"server-negotiator.queue_time_us":54,"threads_started":1}
I20260927 18:45:44.277501 8506 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.010s sys 0.008s
I20260927 18:45:44.277793 8506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "52955f40780d4d3292b5e38fa1532838"
format_stamp: "Formatted at 2026-09-27 18:45:41 on dist-test-slave-rfd5"
server_key: "5d10037f00ebebd9318f195de92b4dd9"
server_key_iv: "eb99993c3454ce1a72b366a05512b891"
server_key_version: "encryptionkey@0"
I20260927 18:45:44.279924 8506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:45:44.309203 8506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:45:44.317312 8506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:45:44.317787 8506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:45:44.319525 8506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:45:44.322239 8506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:45:44.322418 8506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:44.322575 8506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:45:44.322696 8506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:45:44.366153 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:37073 every 8 connection(s)
I20260927 18:45:44.366160 8506 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:37073
I20260927 18:45:44.369189 8506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:45:44.370901 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8506
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:44.403175 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.372166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.31.26.254:34619)
0927 18:45:44.372415 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.372423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.382294 (+ 9871us) server_negotiation.cc:407] Connection header received
0927 18:45:44.382606 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.382614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.382710 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.382855 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.385963 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.387204 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.390202 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.390856 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.391007 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.396087 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:44.396097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:44.396117 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:44.396158 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:44.399350 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:44.400275 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:44.400287 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:44.400297 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:44.400504 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:45:44.400986 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:45:44.400990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:45:44.400992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:45:44.401570 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:45:44.401941 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 18:45:44.402414 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 18:45:44.402827 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 18:45:44.624338 8505 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.136s user 0.007s sys 0.006s
W20260927 18:45:44.625835 8504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:43.475083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:39003 (local address 127.31.26.254:34619)
0927 18:45:43.475747 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:43.475758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:43.476925 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 18:45:43.479355 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:43.479364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:43.479526 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:43.479687 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:43.482352 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:43.484165 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:43.486754 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:43.487173 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:43.487315 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.625475 (+1138160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:39003: BlockingRecv error: recv got EOF from 127.31.26.194:39003 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":282,"threads_started":1}
I20260927 18:45:44.626411 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:43.475759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:39003)
0927 18:45:43.476720 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:43.476775 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:43.478932 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:43.480165 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:43.480181 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:43.481126 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:43.482032 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:43.482061 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:43.484479 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:43.484487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:43.486566 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:43.486581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:43.487425 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:43.487433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:43.487693 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:43.488430 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:43.488454 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.625291 (+1136837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":405,"threads_started":1}
W20260927 18:45:44.627677 8502 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:34619 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.26.254:34619: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:44.647060 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.628766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:53955)
0927 18:45:44.628994 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.629011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.629170 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.629696 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.629701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.630110 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.630274 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.630282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.631322 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.631327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.632263 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.632273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.633008 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.633014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.633239 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.642410 (+ 9171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.642435 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.646830 (+ 4395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 18:45:44.647715 8504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:44.629255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53955 (local address 127.31.26.254:34619)
0927 18:45:44.629428 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.629432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.629453 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:45:44.629500 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.629503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.629560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.629680 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.630424 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.631174 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.632406 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.632819 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.632957 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.647303 (+ 14346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:53955: BlockingRecv error: recv got EOF from 127.31.26.194:53955 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:44 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:44.659792 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.647898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:49017)
0927 18:45:44.649279 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.649296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.649463 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.649832 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.649837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.650476 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.650723 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.650732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.651788 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.651794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.653062 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.653075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.653894 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.653900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.654093 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.654940 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.654959 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:44.659416 (+ 4457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1259,"thread_start_us":1007,"threads_started":1}
W20260927 18:45:44.659989 8504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:44.648293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:49017 (local address 127.31.26.254:34619)
0927 18:45:44.648763 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.648772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.649393 (+ 621us) server_negotiation.cc:407] Connection header received
0927 18:45:44.649628 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.649632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.649696 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.649812 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.650872 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.651631 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.653260 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.653704 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.653849 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.659670 (+ 5821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:49017: BlockingRecv error: recv got EOF from 127.31.26.194:49017 (error 108)
Metrics: {"server-negotiator.queue_time_us":280}
W20260927 18:45:44.660439 8502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:45.287040 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.275214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:54677)
0927 18:45:45.275782 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.275806 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:45.275945 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:45.276386 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:45.276392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:45.276914 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:45.277074 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.277129 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.279234 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.279238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.280281 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.280294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.281782 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.281790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.282040 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.282733 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:45.282757 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:45.286769 (+ 4012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":337,"threads_started":1}
W20260927 18:45:45.287689 8644 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:45.275338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:54677 (local address 127.31.26.254:34619)
0927 18:45:45.276070 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:45.276076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:45.276091 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:45:45.276155 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:45.276158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:45.276223 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:45.276341 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:45.277315 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.278975 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.280453 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.281490 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.281686 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.287209 (+ 5523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:54677: BlockingRecv error: recv got EOF from 127.31.26.193:54677 (error 108)
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":467,"threads_started":1}
W20260927 18:45:45.509748 8640 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.117s user 0.012s sys 0.013s
W20260927 18:45:45.511037 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:44.375680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:41621 (local address 127.31.26.254:34619)
0927 18:45:44.375946 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:44.375956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:44.382836 (+ 6880us) server_negotiation.cc:407] Connection header received
0927 18:45:44.385546 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:44.385552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:44.385649 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:44.385772 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:44.388115 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.388957 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.391241 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.391673 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.391747 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.510749 (+1119002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:41621: BlockingRecv error: recv got EOF from 127.31.26.195:41621 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 18:45:45.511650 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:44.374862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:41621)
0927 18:45:44.382666 (+ 7804us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:44.382729 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:44.385228 (+ 2499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:44.386076 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:44.386093 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:44.386935 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:44.387860 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.387881 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.389206 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.389209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.391059 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:44.391074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:44.392138 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:44.392149 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:44.392392 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:44.393165 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:44.393193 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:45.510570 (+1117377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7364,"thread_start_us":7039,"threads_started":1}
W20260927 18:45:45.512575 8637 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:34619 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.26.254:34619: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:45.522210 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.513309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:56225)
0927 18:45:45.513500 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.513523 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:45.513644 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:45.513969 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:45.513974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:45.514363 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:45.514528 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.514535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.515600 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.515605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.516535 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.516547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.517394 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.517400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.517566 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.518182 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:45.518199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:45.521880 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 18:45:45.522361 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:45.513413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:56225 (local address 127.31.26.254:34619)
0927 18:45:45.513679 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:45.513685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:45.513718 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:45:45.513782 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:45.513786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:45.513843 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:45.513942 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:45.514691 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.515458 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.516691 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.517202 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.517337 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.522100 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:56225: BlockingRecv error: recv got EOF from 127.31.26.195:56225 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:45.531499 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.523068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:59019)
0927 18:45:45.523261 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.523281 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:45.523388 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:45.523725 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:45.523730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:45.524112 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:45.524270 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.524278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.525355 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.525360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.526432 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.526444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.527128 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.527133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.527322 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.527904 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:45.527921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:45.531206 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 18:45:45.531658 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:45.523302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:59019 (local address 127.31.26.254:34619)
0927 18:45:45.523459 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:45.523465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:45.523482 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:45:45.523535 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:45.523539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:45.523590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:45.523695 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:45.524411 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.525223 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.526569 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.526969 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.527095 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.531419 (+ 4324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:59019: BlockingRecv error: recv got EOF from 127.31.26.195:59019 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 18:45:45.531868 8637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:45 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:45.669767 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:45.661002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:49721)
0927 18:45:45.661598 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:45.661614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:45.661734 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:45.662052 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:45.662055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:45.662421 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:45.662570 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.662579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.663601 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.663606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.664531 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:45.664543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.665279 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.665284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:45.665409 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.666079 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:45.666097 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:45.669535 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":385,"threads_started":1}
W20260927 18:45:45.670054 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:45.661169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:49721 (local address 127.31.26.254:34619)
0927 18:45:45.661342 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:45.661348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:45.661711 (+ 363us) server_negotiation.cc:407] Connection header received
0927 18:45:45.661840 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:45.661844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:45.661926 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:45.662011 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:45.662734 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.663469 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.664651 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:45.665107 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:45.665199 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:45.669756 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:49721: BlockingRecv error: recv got EOF from 127.31.26.194:49721 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 18:45:46 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:46 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:46.297371 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:46.288074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:58461)
0927 18:45:46.288676 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:46.288695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:46.288815 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:46.289150 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:46.289153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:46.289624 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:46.289803 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:46.289812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.290907 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.290911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:46.291888 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:46.291903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.292724 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.292729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:46.292884 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:46.293553 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:46.293570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:46.297060 (+ 3490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
W20260927 18:45:46.297627 8647 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:46.288173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:58461 (local address 127.31.26.254:34619)
0927 18:45:46.288708 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:46.288712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:46.288756 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:45:46.288941 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:46.288945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:46.289004 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:46.289124 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:46.289975 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.290787 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.292029 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.292555 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.292680 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:46.297328 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:58461: BlockingRecv error: recv got EOF from 127.31.26.193:58461 (error 108)
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
Sep 27 18:45:46 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:46 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:46.542920 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:46.532559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:42771)
0927 18:45:46.533205 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:46.533225 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:46.533381 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:46.533852 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:46.533856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:46.534413 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:46.534627 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:46.534653 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.535807 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.535811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:46.536865 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:46.536878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.537720 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.537733 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:46.537867 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:46.538541 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:46.538565 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:46.542644 (+ 4079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":335,"threads_started":1}
W20260927 18:45:46.543068 8647 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:46.532798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:42771 (local address 127.31.26.254:34619)
0927 18:45:46.533048 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:46.533054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:46.533373 (+ 319us) server_negotiation.cc:407] Connection header received
0927 18:45:46.533570 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:46.533585 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:46.533687 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:46.533780 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:46.534842 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.535642 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.537071 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.537518 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.537626 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:46.542842 (+ 5216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:42771: BlockingRecv error: recv got EOF from 127.31.26.195:42771 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 18:45:46 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:46 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:46.680748 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:46.670896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:49407)
0927 18:45:46.671427 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:46.671443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:46.671625 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:46.671985 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:46.671990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:46.672411 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:46.672577 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:46.672586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.673806 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.673812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:46.674773 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:46.674785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.675754 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.675760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:46.675957 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:46.676702 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:46.676748 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:46.680487 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
W20260927 18:45:46.681022 8647 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:46.671050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:49407 (local address 127.31.26.254:34619)
0927 18:45:46.671297 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:46.671304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:46.671552 (+ 248us) server_negotiation.cc:407] Connection header received
0927 18:45:46.671753 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:46.671759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:46.671828 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:46.671952 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:46.672735 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.673585 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.674921 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:46.675543 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:46.675726 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:46.680703 (+ 4977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:49407: BlockingRecv error: recv got EOF from 127.31.26.194:49407 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 18:45:47 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:47 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:47.307665 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:47.298488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:44035)
0927 18:45:47.298998 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:47.299016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:47.299128 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:47.299724 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:47.299729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:47.300208 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:47.300429 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:47.300441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.301581 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.301585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:47.302515 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:47.302528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.303243 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.303247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:47.303373 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:47.303996 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:47.304014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:47.307429 (+ 3415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
W20260927 18:45:47.308116 8651 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:47.298710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:44035 (local address 127.31.26.254:34619)
0927 18:45:47.299334 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:47.299339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:47.299375 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:45:47.299443 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:47.299448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:47.299523 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:47.299637 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:47.300575 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.301429 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.302649 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.303071 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.303196 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:47.307667 (+ 4471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:44035: BlockingRecv error: recv got EOF from 127.31.26.193:44035 (error 108)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":378,"threads_started":1}
Sep 27 18:45:47 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:47 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:47.553478 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:47.543966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:36639)
0927 18:45:47.544508 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:47.544524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:47.544688 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:47.545048 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:47.545053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:47.545491 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:47.545679 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:47.545688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.547011 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.547024 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:47.548108 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:47.548120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.548813 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.548818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:47.549000 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:47.549653 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:47.549671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:47.553222 (+ 3551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":326,"threads_started":1}
W20260927 18:45:47.553670 8651 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:47.544039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:36639 (local address 127.31.26.254:34619)
0927 18:45:47.544316 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:47.544325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:47.544617 (+ 292us) server_negotiation.cc:407] Connection header received
0927 18:45:47.544838 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:47.544842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:47.544902 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:47.545020 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:47.545825 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.546858 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.548251 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.548662 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.548802 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:47.553397 (+ 4595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:36639: BlockingRecv error: recv got EOF from 127.31.26.195:36639 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 18:45:47 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:47 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:47.691002 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:47.681767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:49405)
0927 18:45:47.682299 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:47.682315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:47.682432 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:47.682756 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:47.682760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:47.683148 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:47.683405 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:47.683416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.684885 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.684892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:47.685871 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:47.685883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.686551 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.686555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:47.686694 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:47.687285 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:47.687302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:47.690775 (+ 3473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":278,"threads_started":1}
W20260927 18:45:47.691200 8651 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:47.681837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:49405 (local address 127.31.26.254:34619)
0927 18:45:47.682006 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:47.682010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:47.682388 (+ 378us) server_negotiation.cc:407] Connection header received
0927 18:45:47.682529 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:47.682533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:47.682607 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:47.682700 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:47.683569 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.684721 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.685988 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:47.686428 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:47.686521 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:47.690957 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:49405: BlockingRecv error: recv got EOF from 127.31.26.194:49405 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 18:45:48 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:48 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:48.318759 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:48.308746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:43279)
0927 18:45:48.309609 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:48.309626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:48.309747 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:48.310072 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:48.310076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:48.310446 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:48.310607 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:48.310615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.311793 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.311797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:48.312907 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:48.312920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.314007 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.314016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:48.314261 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:48.315003 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:48.315030 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:48.318500 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":655,"threads_started":1}
W20260927 18:45:48.318979 8655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:48.308856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:43279 (local address 127.31.26.254:34619)
0927 18:45:48.309499 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:48.309505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:48.309718 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:45:48.309852 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:48.309856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:48.309917 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:48.310011 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:48.310793 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.311641 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.313222 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.313824 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.313951 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:48.318719 (+ 4768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:43279: BlockingRecv error: recv got EOF from 127.31.26.193:43279 (error 108)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":366,"threads_started":1}
Sep 27 18:45:48 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:48 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:48.563526 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:48.554500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:48349)
0927 18:45:48.555026 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:48.555043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:48.555162 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:48.555451 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:48.555455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:48.555797 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:48.555984 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:48.555993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.556990 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.556995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:48.557947 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:48.557967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.558616 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.558620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:48.558793 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:48.559408 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:48.559426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:48.563206 (+ 3780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
W20260927 18:45:48.563666 8655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:48.554610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:48349 (local address 127.31.26.254:34619)
0927 18:45:48.554873 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:48.554879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:48.555123 (+ 244us) server_negotiation.cc:407] Connection header received
0927 18:45:48.555258 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:48.555262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:48.555325 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:48.555409 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:48.556114 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.556838 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.558092 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.558462 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.558554 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:48.563399 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:48349: BlockingRecv error: recv got EOF from 127.31.26.195:48349 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 18:45:48 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:48 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:48.701787 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:48.692008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:49417)
0927 18:45:48.692527 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:48.692543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:48.692655 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:48.692957 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:48.692960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:48.693384 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:48.693541 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:48.693550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.694783 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.694788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:48.696065 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:48.696081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.696833 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.696837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:48.696970 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:48.697632 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:48.697652 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:48.701483 (+ 3831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
W20260927 18:45:48.702077 8655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:48.692132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:49417 (local address 127.31.26.254:34619)
0927 18:45:48.692391 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:48.692398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:48.692656 (+ 258us) server_negotiation.cc:407] Connection header received
0927 18:45:48.692741 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:48.692745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:48.692826 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:48.692914 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:48.693685 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.694599 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.696217 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:48.696674 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:48.696778 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:48.701795 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:49417: BlockingRecv error: recv got EOF from 127.31.26.194:49417 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 27 18:45:49 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:49 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:49.328933 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:49.319781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:45561)
0927 18:45:49.320372 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:49.320388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:49.320501 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:49.320799 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:49.320802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:49.321276 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:49.321426 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:49.321434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.322424 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.322428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:49.323459 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:49.323472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.324175 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.324181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:49.324369 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:49.325051 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:49.325069 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:49.328642 (+ 3573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":335,"threads_started":1}
W20260927 18:45:49.329293 8659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:49.319847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:45561 (local address 127.31.26.254:34619)
0927 18:45:49.320376 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:49.320381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:49.320464 (+ 83us) server_negotiation.cc:407] Connection header received
0927 18:45:49.320595 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:49.320598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:49.320670 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:49.320759 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:49.321553 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.322284 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.323620 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.324025 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.324163 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:49.328930 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:45561: BlockingRecv error: recv got EOF from 127.31.26.193:45561 (error 108)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":329,"threads_started":1}
Sep 27 18:45:49 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:49 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:49.574234 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:49.564682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:47505)
0927 18:45:49.565253 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:49.565269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:49.565430 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:49.565774 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:49.565779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:49.566226 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:49.566438 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:49.566450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.567755 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.567763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:49.568968 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:49.568981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.569661 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.569665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:49.569789 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:49.570373 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:49.570393 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:49.573953 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":304,"threads_started":1}
W20260927 18:45:49.574412 8659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:49.564788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:47505 (local address 127.31.26.254:34619)
0927 18:45:49.565037 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:49.565047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:49.565361 (+ 314us) server_negotiation.cc:407] Connection header received
0927 18:45:49.565555 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:49.565561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:49.565631 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:49.565746 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:49.566647 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.567555 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.569133 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.569527 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.569637 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:49.574164 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:47505: BlockingRecv error: recv got EOF from 127.31.26.195:47505 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 18:45:49 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:49 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:49.718194 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:49.702799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:56297)
0927 18:45:49.703368 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:49.703390 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:49.703585 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:49.703998 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:49.704003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:49.704519 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:49.704727 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:49.704737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.705866 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.705872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:49.706829 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:49.706840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.707580 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.707586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:49.707786 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:49.713985 (+ 6199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:49.714019 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:49.717963 (+ 3944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":292,"threads_started":1}
W20260927 18:45:49.718402 8659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:49.703123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:56297 (local address 127.31.26.254:34619)
0927 18:45:49.703349 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:49.703355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:49.703486 (+ 131us) server_negotiation.cc:407] Connection header received
0927 18:45:49.703746 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:49.703751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:49.703834 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:49.703964 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:49.704876 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.705730 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.706989 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:49.707416 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:49.707559 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:49.718149 (+ 10590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:56297: BlockingRecv error: recv got EOF from 127.31.26.194:56297 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 18:45:50 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:50 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:50.339267 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:50.329940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:51029)
0927 18:45:50.330464 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:50.330482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:50.330591 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:50.331237 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:50.331242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:50.331617 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:50.331793 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:50.331801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.332829 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.332832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:50.333871 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:50.333883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.334576 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.334580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:50.334699 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:50.335367 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:50.335386 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:50.338910 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":290,"threads_started":1}
W20260927 18:45:50.339358 8663 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:50.330003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:51029 (local address 127.31.26.254:34619)
0927 18:45:50.330957 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:50.330961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:50.330975 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:45:50.331019 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:50.331022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:50.331084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:50.331176 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:50.331962 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.332706 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.334021 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.334431 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.334539 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:50.339095 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:51029: BlockingRecv error: recv got EOF from 127.31.26.193:51029 (error 108)
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":748,"threads_started":1}
Sep 27 18:45:50 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:50 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:50.584852 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:50.575300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:42939)
0927 18:45:50.575931 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:50.575947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:50.576118 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:50.576522 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:50.576528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:50.577042 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:50.577225 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:50.577233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.578279 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.578283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:50.579238 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:50.579250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.579971 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.579976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:50.580111 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:50.580799 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:50.580817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:50.584621 (+ 3804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":392,"threads_started":1}
W20260927 18:45:50.585222 8663 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:50.575366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:42939 (local address 127.31.26.254:34619)
0927 18:45:50.575575 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:50.575581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:50.576074 (+ 493us) server_negotiation.cc:407] Connection header received
0927 18:45:50.576262 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:50.576268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:50.576352 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:50.576463 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:50.577364 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.578115 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.579391 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.579801 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.579906 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:50.584971 (+ 5065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:42939: BlockingRecv error: recv got EOF from 127.31.26.195:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 18:45:50 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:50 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:50.728535 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:50.719190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:60951)
0927 18:45:50.719733 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:50.719750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:50.719919 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:50.720272 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:50.720276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:50.720735 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:50.720896 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:50.720904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.721992 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.722000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:50.722921 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:50.722933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.723641 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.723646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:50.723836 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:50.724433 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:50.724450 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:50.728311 (+ 3861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
W20260927 18:45:50.728786 8663 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:50.719354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:60951 (local address 127.31.26.254:34619)
0927 18:45:50.719615 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:50.719621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:50.719846 (+ 225us) server_negotiation.cc:407] Connection header received
0927 18:45:50.720044 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:50.720052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:50.720129 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:50.720237 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:50.721074 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.721858 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.723066 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:50.723476 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:50.723613 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:50.728499 (+ 4886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:60951: BlockingRecv error: recv got EOF from 127.31.26.194:60951 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 18:45:51 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:51 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:51.351419 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:51.340417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:37791)
0927 18:45:51.342317 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:51.342343 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:51.342499 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:51.342905 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:51.342911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:51.343355 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:51.343530 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:51.343540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.344748 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.344753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:51.345864 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:51.345880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.346621 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.346625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:51.346748 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:51.347332 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:51.347350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:51.351027 (+ 3677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1754,"thread_start_us":1509,"threads_started":1}
W20260927 18:45:51.351747 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:51.340484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:37791 (local address 127.31.26.254:34619)
0927 18:45:51.342586 (+ 2102us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:51.342594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:51.342613 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:45:51.342675 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:51.342678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:51.342741 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:51.342870 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:51.343718 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.344498 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.346024 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.346469 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.346574 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:51.351299 (+ 4725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:37791: BlockingRecv error: recv got EOF from 127.31.26.193:37791 (error 108)
Metrics: {"server-negotiator.queue_time_us":1976,"thread_start_us":1836,"threads_started":1}
Sep 27 18:45:51 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:51 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:51.595402 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:51.585891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:36333)
0927 18:45:51.586472 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:51.586488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:51.586669 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:51.587014 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:51.587019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:51.587582 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:51.587813 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:51.587823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.589030 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.589035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:51.590106 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:51.590119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.590885 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.590891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:51.591076 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:51.591669 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:51.591686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:51.595143 (+ 3457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":317,"threads_started":1}
W20260927 18:45:51.595623 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:51.586092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:36333 (local address 127.31.26.254:34619)
0927 18:45:51.586351 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:51.586357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:51.586589 (+ 232us) server_negotiation.cc:407] Connection header received
0927 18:45:51.586812 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:51.586816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:51.586876 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:51.586984 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:51.587959 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.588891 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.590264 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.590720 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.590846 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:51.595366 (+ 4520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:36333: BlockingRecv error: recv got EOF from 127.31.26.195:36333 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 18:45:51 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:51 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:51.738898 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:51.729502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:45219)
0927 18:45:51.730137 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:51.730154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:51.730270 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:51.730620 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:51.730624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:51.731038 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:51.731230 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:51.731239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.732268 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.732272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:51.733255 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:51.733268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.733976 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.733980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:51.734102 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:51.734720 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:51.734737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:51.738526 (+ 3789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":299,"threads_started":1}
W20260927 18:45:51.739068 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:51.729575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:45219 (local address 127.31.26.254:34619)
0927 18:45:51.729769 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:51.729773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:51.730235 (+ 462us) server_negotiation.cc:407] Connection header received
0927 18:45:51.730371 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:51.730375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:51.730437 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:51.730520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:51.731376 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.732128 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.733390 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:51.733830 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:51.733930 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:51.738746 (+ 4816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:45219: BlockingRecv error: recv got EOF from 127.31.26.194:45219 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:45:52 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:52 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:52.361840 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:52.352322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:37025)
0927 18:45:52.352952 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:52.352973 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:52.353176 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:52.353556 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:52.353571 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:52.353957 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:52.354123 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:52.354131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.355228 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.355232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:52.356164 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:52.356174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.356915 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.356920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:52.357030 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:52.357722 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:52.357740 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:52.361503 (+ 3763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":329,"threads_started":1}
W20260927 18:45:52.361968 8671 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:52.352403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:37025 (local address 127.31.26.254:34619)
0927 18:45:52.353265 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:52.353269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:52.353285 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:45:52.353326 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:52.353329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:52.353392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:52.353480 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:52.354283 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.355089 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.356297 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.356762 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.356861 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:52.361711 (+ 4850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:37025: BlockingRecv error: recv got EOF from 127.31.26.193:37025 (error 108)
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":638,"threads_started":1}
Sep 27 18:45:52 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:52 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:52.606014 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:52.596517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:58751)
0927 18:45:52.597231 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:52.597248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:52.597362 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:52.597651 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:52.597654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:52.598053 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:52.598217 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:52.598225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.599222 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.599227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:52.600314 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:52.600334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.601077 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.601157 (+ 80us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:52.601300 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:52.601933 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:52.601964 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:52.605648 (+ 3684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":327,"threads_started":1}
W20260927 18:45:52.606068 8671 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:52.596624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:58751 (local address 127.31.26.254:34619)
0927 18:45:52.596846 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:52.596855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:52.597323 (+ 468us) server_negotiation.cc:407] Connection header received
0927 18:45:52.597461 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:52.597465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:52.597527 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:52.597608 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:52.598340 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.599067 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.600498 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.600930 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.601033 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:52.605840 (+ 4807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:58751: BlockingRecv error: recv got EOF from 127.31.26.195:58751 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 18:45:52 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:52 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:52.749964 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:52.740213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:40891)
0927 18:45:52.740797 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:52.740814 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:52.740952 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:52.741283 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:52.741286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:52.741681 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:52.741862 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:52.741871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.743100 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.743104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:52.744084 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:52.744098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.744905 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.744911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:52.745150 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:52.745839 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:52.745856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:52.749610 (+ 3754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
W20260927 18:45:52.750118 8671 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:52.740369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:40891 (local address 127.31.26.254:34619)
0927 18:45:52.740669 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:52.740675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:52.740938 (+ 263us) server_negotiation.cc:407] Connection header received
0927 18:45:52.741047 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:52.741051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:52.741143 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:52.741239 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:52.742007 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.742907 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.744271 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:52.744703 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:52.744846 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:52.749842 (+ 4996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:40891: BlockingRecv error: recv got EOF from 127.31.26.194:40891 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 27 18:45:53 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:53 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:53.372740 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:53.362864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:37365)
0927 18:45:53.363325 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:53.363341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:53.363445 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:53.363971 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:53.363974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:53.364387 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:53.364532 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:53.364540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.365639 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.365642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:53.366702 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:53.366716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.367712 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.367716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:53.367847 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:53.368569 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:53.368593 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:53.372505 (+ 3912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
W20260927 18:45:53.373209 8675 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:53.362985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:37365 (local address 127.31.26.254:34619)
0927 18:45:53.363692 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:53.363697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:53.363714 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:45:53.363763 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:53.363766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:53.363829 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:53.363924 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:53.364695 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.365497 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.366858 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.367536 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.367682 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:53.372722 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:37365: BlockingRecv error: recv got EOF from 127.31.26.193:37365 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":483,"threads_started":1}
Sep 27 18:45:53 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:53 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:53.616962 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:53.607047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:35337)
0927 18:45:53.607875 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:53.607892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:53.608010 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:53.608351 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:53.608355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:53.608771 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:53.608937 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:53.608946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.609994 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.609998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:53.610984 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:53.610997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.611725 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.611730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:53.611856 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:53.612526 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:53.612546 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:53.616527 (+ 3981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":527,"threads_started":1}
W20260927 18:45:53.617040 8675 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:53.607178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:35337 (local address 127.31.26.254:34619)
0927 18:45:53.607414 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:53.607423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:53.607974 (+ 551us) server_negotiation.cc:407] Connection header received
0927 18:45:53.608129 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:53.608133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:53.608203 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:53.608290 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:53.609069 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.609837 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.611141 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.611544 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.611645 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:53.616771 (+ 5126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:35337: BlockingRecv error: recv got EOF from 127.31.26.195:35337 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 18:45:53 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:53 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:53.761253 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:53.751143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:47951)
0927 18:45:53.751717 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:53.751739 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:53.751872 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:53.752231 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:53.752234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:53.752676 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:53.752837 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:53.752847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.754035 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.754040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:53.754986 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:53.755009 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.756057 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.756064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:53.756266 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:53.757121 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:53.757176 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:53.760946 (+ 3770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":346,"threads_started":1}
W20260927 18:45:53.761458 8675 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:53.751226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:47951 (local address 127.31.26.254:34619)
0927 18:45:53.751505 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:53.751516 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:53.751844 (+ 328us) server_negotiation.cc:407] Connection header received
0927 18:45:53.751985 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:53.751989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:53.752071 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:53.752183 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:53.752984 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.753835 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.755156 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:53.755864 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:53.755990 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:53.761220 (+ 5230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:47951: BlockingRecv error: recv got EOF from 127.31.26.194:47951 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:45:54 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:54 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:54.383785 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:54.373904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:57849)
0927 18:45:54.374532 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:54.374552 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:54.374688 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:54.375084 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:54.375089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:54.375590 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:54.375773 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:54.375783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.377014 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.377023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:54.378134 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:54.378148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.378947 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.378954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:54.379142 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:54.379852 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:54.379879 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:54.383522 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":346,"threads_started":1}
W20260927 18:45:54.383965 8679 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:54.374024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:57849 (local address 127.31.26.254:34619)
0927 18:45:54.374523 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:54.374527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:54.374645 (+ 118us) server_negotiation.cc:407] Connection header received
0927 18:45:54.374797 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:54.374801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:54.374877 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:54.375044 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:54.375954 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.376859 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.378299 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.378771 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.378928 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:54.383726 (+ 4798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:57849: BlockingRecv error: recv got EOF from 127.31.26.193:57849 (error 108)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":307,"threads_started":1}
Sep 27 18:45:54 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:54 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:54.627529 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:54.618039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:47373)
0927 18:45:54.618586 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:54.618616 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:54.618767 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:54.619096 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:54.619101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:54.619519 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:54.619669 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:54.619677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.620701 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.620709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:54.621904 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:54.621916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.622626 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.622632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:54.622806 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:54.623384 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:54.623401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:54.627177 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":327,"threads_started":1}
W20260927 18:45:54.627588 8679 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:54.618086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:47373 (local address 127.31.26.254:34619)
0927 18:45:54.618269 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:54.618274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:54.618705 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:45:54.618889 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:54.618895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:54.618961 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:54.619069 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:54.619816 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.620559 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.622057 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.622459 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.622597 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:54.627346 (+ 4749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:47373: BlockingRecv error: recv got EOF from 127.31.26.195:47373 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 18:45:54 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:54 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:54.777787 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:54.762435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:46645)
0927 18:45:54.763460 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:54.763482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:54.763600 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:54.763998 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:54.764001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:54.764486 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:54.764655 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:54.764665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.765784 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.765788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:54.766772 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:54.766783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.767460 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.767464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:54.767582 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:54.773585 (+ 6003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:54.773619 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:54.777557 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":797,"threads_started":1}
W20260927 18:45:54.777945 8679 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:54.762530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:46645 (local address 127.31.26.254:34619)
0927 18:45:54.762701 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:54.762705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:54.763548 (+ 843us) server_negotiation.cc:407] Connection header received
0927 18:45:54.763738 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:54.763744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:54.763853 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:54.763956 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:54.764807 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.765659 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.766906 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:54.767323 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:54.767426 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:54.777707 (+ 10281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:46645: BlockingRecv error: recv got EOF from 127.31.26.194:46645 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 18:45:55 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:55 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:55.393585 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:55.384638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:42441)
0927 18:45:55.385209 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:55.385233 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:55.385337 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:55.385627 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:55.385630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:55.386014 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:55.386191 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:55.386200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.387219 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.387223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:55.388166 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:55.388178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.388900 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.388904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:55.389019 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:55.389724 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:55.389742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:55.393248 (+ 3506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
W20260927 18:45:55.393782 8683 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:55.384800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:42441 (local address 127.31.26.254:34619)
0927 18:45:55.385286 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:55.385290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:55.385306 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:45:55.385438 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:55.385441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:55.385499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:55.385587 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:55.386306 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.387105 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.388309 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.388762 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.388926 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:55.393468 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:42441: BlockingRecv error: recv got EOF from 127.31.26.193:42441 (error 108)
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":250,"threads_started":1}
Sep 27 18:45:55 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:55 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:55.638262 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:55.628707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:42353)
0927 18:45:55.629270 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:55.629305 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:55.629461 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:55.629942 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:55.629946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:55.630423 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:55.630634 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:55.630666 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.631754 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.631758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:55.632772 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:55.632784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.633427 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.633431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:55.633557 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:55.634148 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:55.634166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:55.637877 (+ 3711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":274,"threads_started":1}
W20260927 18:45:55.638316 8683 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:55.628787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:42353 (local address 127.31.26.254:34619)
0927 18:45:55.629007 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:55.629016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:55.629403 (+ 387us) server_negotiation.cc:407] Connection header received
0927 18:45:55.629608 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:55.629611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:55.629674 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:55.629755 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:55.630804 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.631627 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.632905 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.633308 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.633397 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:55.638071 (+ 4674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:42353: BlockingRecv error: recv got EOF from 127.31.26.195:42353 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:45:55 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:55 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:55.787588 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:55.778898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:32951)
0927 18:45:55.779397 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:55.779420 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:55.779531 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:55.779823 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:55.779826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:55.780236 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:55.780405 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:55.780413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.781444 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.781448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:55.782366 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:55.782377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.783249 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.783254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:55.783371 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:55.784081 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:55.784111 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:55.787373 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":298,"threads_started":1}
W20260927 18:45:55.787832 8683 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:55.779114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:32951 (local address 127.31.26.254:34619)
0927 18:45:55.779272 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:55.779276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:55.779491 (+ 215us) server_negotiation.cc:407] Connection header received
0927 18:45:55.779622 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:55.779626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:55.779700 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:55.779782 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:55.780531 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.781308 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.782487 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:55.783106 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:55.783209 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:55.787571 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:32951: BlockingRecv error: recv got EOF from 127.31.26.194:32951 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 18:45:56 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:56 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:56.404196 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:56.394912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:52383)
0927 18:45:56.395719 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:56.395740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:56.395883 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:56.396235 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:56.396240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:56.396715 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:56.396877 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:56.396886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.397952 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.397956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:56.398876 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:56.398886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.399552 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.399557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:56.399699 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:56.400432 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:56.400450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:56.403818 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":527,"threads_started":1}
W20260927 18:45:56.404269 8687 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:56.394971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:52383 (local address 127.31.26.254:34619)
0927 18:45:56.395701 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:56.395705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:56.395833 (+ 128us) server_negotiation.cc:407] Connection header received
0927 18:45:56.395995 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:56.396002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:56.396086 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:56.396172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:56.397001 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.397800 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.398997 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.399401 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.399494 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:56.404015 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:52383: BlockingRecv error: recv got EOF from 127.31.26.193:52383 (error 108)
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":321,"threads_started":1}
Sep 27 18:45:56 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:56 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:56.648360 8688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:56.639280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:51331)
0927 18:45:56.639786 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:56.639811 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:56.639972 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:56.640311 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:56.640315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:56.640754 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:56.640975 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:56.640988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.642305 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.642311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:56.643432 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:56.643444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.644133 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.644138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:56.644259 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:56.644860 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:56.644876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:56.648143 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":290,"threads_started":1}
W20260927 18:45:56.648543 8687 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:56.639402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:51331 (local address 127.31.26.254:34619)
0927 18:45:56.639647 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:56.639656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:56.639895 (+ 239us) server_negotiation.cc:407] Connection header received
0927 18:45:56.640116 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:56.640120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:56.640179 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:56.640282 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:56.641228 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.642178 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.643559 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.643969 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.644066 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:56.648328 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:51331: BlockingRecv error: recv got EOF from 127.31.26.195:51331 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:45:56 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:56 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:56.798401 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:56.788683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:53169)
0927 18:45:56.789315 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:56.789335 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:56.789506 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:56.789892 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:56.789895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:56.790440 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:56.790625 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:56.790636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.791640 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.791644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:56.792798 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:56.792811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.793578 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.793582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:56.793720 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:56.794450 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:56.794467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:56.798158 (+ 3691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":299,"threads_started":1}
W20260927 18:45:56.798655 8687 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:56.788749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53169 (local address 127.31.26.254:34619)
0927 18:45:56.788947 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:56.788955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:56.789421 (+ 466us) server_negotiation.cc:407] Connection header received
0927 18:45:56.789676 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:56.789680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:56.789745 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:56.789853 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:56.790766 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.791498 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.792955 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:56.793414 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:56.793521 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:56.798361 (+ 4840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:53169: BlockingRecv error: recv got EOF from 127.31.26.194:53169 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 18:45:57 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:57 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:57.413866 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:57.405075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:46679)
0927 18:45:57.405539 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:57.405556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:57.405688 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:57.406300 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:57.406303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:57.406693 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:57.406841 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:57.406849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.407765 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.407769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:57.408716 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:57.408727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.409420 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.409425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:57.409562 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:57.410152 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:57.410169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:57.413517 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":221,"threads_started":1}
W20260927 18:45:57.414076 8691 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:57.405192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:46679 (local address 127.31.26.254:34619)
0927 18:45:57.406048 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:57.406052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:57.406067 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:45:57.406105 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:57.406108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:57.406171 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:57.406255 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:57.406961 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.407652 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.408846 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.409280 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.409370 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:57.413794 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:46679: BlockingRecv error: recv got EOF from 127.31.26.193:46679 (error 108)
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":665,"threads_started":1}
Sep 27 18:45:57 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:57 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:57.658470 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:57.649545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:36063)
0927 18:45:57.650117 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:57.650133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:57.650290 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:57.650648 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:57.650652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:57.651123 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:57.651281 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:57.651290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.652334 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.652341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:57.653288 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:57.653298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.654034 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.654040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:57.654211 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:57.654784 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:57.654801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:57.658224 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":331,"threads_started":1}
W20260927 18:45:57.658814 8691 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:57.649621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:36063 (local address 127.31.26.254:34619)
0927 18:45:57.649863 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:57.649871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:57.650221 (+ 350us) server_negotiation.cc:407] Connection header received
0927 18:45:57.650440 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:57.650444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:57.650503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:57.650620 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:57.651435 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.652182 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.653429 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.653846 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.653977 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:57.658441 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:36063: BlockingRecv error: recv got EOF from 127.31.26.195:36063 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 18:45:57 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:57 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:57.808404 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:57.799311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:53771)
0927 18:45:57.799809 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:57.799825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:57.799968 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:57.800306 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:57.800312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:57.800742 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:57.800889 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:57.800897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.802025 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.802031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:57.802961 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:57.802973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.803766 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.803772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:57.803977 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:57.804682 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:57.804701 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:57.808093 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
W20260927 18:45:57.808554 8691 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:57.799458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53771 (local address 127.31.26.254:34619)
0927 18:45:57.799681 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:57.799687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:57.799903 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:45:57.800113 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:57.800118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:57.800176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:57.800274 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:57.801033 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.801885 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.803116 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:57.803619 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:57.803749 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:57.808308 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:53771: BlockingRecv error: recv got EOF from 127.31.26.194:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 18:45:58 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:58 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.193@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:58.424091 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:58.415068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.193:34143)
0927 18:45:58.415633 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:58.415658 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:58.415763 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:58.416104 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:58.416107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:58.416460 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:58.416642 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:58.416650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.417696 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.417699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:58.418606 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:58.418616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.419321 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.419327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:58.419498 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:58.420268 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:58.420286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:58.423842 (+ 3556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":234,"threads_started":1}
W20260927 18:45:58.424275 8695 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:58.415264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:34143 (local address 127.31.26.254:34619)
0927 18:45:58.415721 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:58.415726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:58.415743 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:45:58.415869 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:58.415872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:58.415932 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:58.416057 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:58.416769 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.417575 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.418723 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.419163 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.419257 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:58.424045 (+ 4788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.193:34143: BlockingRecv error: recv got EOF from 127.31.26.193:34143 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":259,"threads_started":1}
Sep 27 18:45:58 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:58 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.195@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:58.669404 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:58.659484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.195:33297)
0927 18:45:58.660073 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:58.660091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:58.660260 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:58.660644 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:58.660651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:58.661183 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:58.661357 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:58.661367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.662437 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.662444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:58.663461 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:58.663474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.664278 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.664284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:58.664482 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:58.665194 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:58.665213 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:58.669047 (+ 3834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":282,"threads_started":1}
W20260927 18:45:58.669606 8695 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:58.659584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:33297 (local address 127.31.26.254:34619)
0927 18:45:58.659978 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:58.659986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:58.660179 (+ 193us) server_negotiation.cc:407] Connection header received
0927 18:45:58.660414 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:58.660419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:58.660482 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:58.660605 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:58.661530 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.662272 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.663635 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.664109 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.664260 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:58.669326 (+ 5066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.195:33297: BlockingRecv error: recv got EOF from 127.31.26.195:33297 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
Sep 27 18:45:58 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:45:58 dist-test-slave-rfd5 krb5kdc[7633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.26.194@KRBTEST.COM for oryx/127.31.26.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:45:58.819674 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:58.809630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:34619 (local address 127.31.26.194:53897)
0927 18:45:58.810206 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 18:45:58.810236 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:45:58.810534 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:45:58.811023 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:45:58.811027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:45:58.811500 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:45:58.811791 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:58.811803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.812965 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.812969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:58.814005 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:45:58.814016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.814810 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.814816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:45:58.815012 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:58.815700 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:45:58.815726 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:45:58.819425 (+ 3699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.26.254:34619: Server oryx/127.31.26.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
W20260927 18:45:58.819861 8695 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:45:58.809788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53897 (local address 127.31.26.254:34619)
0927 18:45:58.810011 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:58.810018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:58.810449 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:45:58.810691 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:58.810695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:58.810791 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:58.810893 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:58.811967 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.812781 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.814165 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:58.814669 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:58.814764 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:58.819605 (+ 4841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.26.194:53897: BlockingRecv error: recv got EOF from 127.31.26.194:53897 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:45:59.377450 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8222
W20260927 18:45:59.397322 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8222: Not authorized: Client connection negotiation failed: client connection to 127.31.26.193:41981: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:45:59.397465 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:59.378741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.31.26.193:41981)
0927 18:45:59.381707 (+ 2966us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:59.381720 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:59.382194 (+ 474us) server_negotiation.cc:407] Connection header received
0927 18:45:59.382348 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:59.382360 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:59.382733 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:59.382986 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:59.384440 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:59.386515 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:59.388057 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:59.389300 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:59.389475 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:59.392374 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:59.392387 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:59.392414 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:59.392477 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:59.396551 (+ 4074us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:45:59.397011 (+ 460us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48172: Request ticket server kudu/127.31.26.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":2712,"thread_start_us":2544,"threads_started":1}
I20260927 18:45:59.397511 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8222
I20260927 18:45:59.414974 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8373
I20260927 18:45:59.432001 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:59.416161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.31.26.194:35471)
0927 18:45:59.417013 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:59.417025 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:59.417061 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:45:59.417189 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:59.417201 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:59.417554 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:59.417805 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:59.419625 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:59.421633 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:59.422958 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:59.424113 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:59.424278 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:59.428177 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:59.428193 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:59.428215 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:59.428297 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:59.431409 (+ 3112us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:45:59.431742 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42198: Request ticket server kudu/127.31.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":479,"threads_started":1}
W20260927 18:45:59.432273 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8373: Not authorized: Client connection negotiation failed: client connection to 127.31.26.194:35471: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.26.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:45:59.432498 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8373
I20260927 18:45:59.449422 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8506
I20260927 18:45:59.468992 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:45:59.450744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.31.26.195:37073)
0927 18:45:59.455999 (+ 5255us) server_negotiation.cc:206] Beginning negotiation
0927 18:45:59.456010 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:45:59.456051 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:45:59.456161 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:45:59.456169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:45:59.456436 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:45:59.456692 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:45:59.458374 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:59.459521 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:59.460931 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:45:59.461669 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:45:59.461839 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:45:59.464860 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:45:59.464870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:45:59.464885 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:45:59.464937 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:45:59.468097 (+ 3160us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:45:59.468550 (+ 453us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34216: Request ticket server kudu/127.31.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":5038,"thread_start_us":4890,"threads_started":1}
W20260927 18:45:59.468995 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8506: Not authorized: Client connection negotiation failed: client connection to 127.31.26.195:37073: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:45:59.469153 8701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34216: Request ticket server kudu/127.31.26.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:45:59.469205 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8506
I20260927 18:45:59.488607 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8153
I20260927 18:45:59.490072 8215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:45:59.603582 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8153
2026-09-27T18:45:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21820 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T18:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:45:59Z Disabled control of system clock
I20260927 18:45:59.680663 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35839
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:46:00.016168 8706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:00.016471 8706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:00.016549 8706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:00.023947 8706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:00.024060 8706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:00.024098 8706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:00.024149 8706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:00.024197 8706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:00.033517 8706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:00.034783 8706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:00.036522 8706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:00.049142 8712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.049183 8714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.049685 8706 server_base.cc:1034] running on GCE node
W20260927 18:46:00.049888 8711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.050850 8706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:00.052466 8706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:00.053673 8706 hybrid_clock.cc:648] HybridClock initialized: now 1790534760053627 us; error 36 us; skew 500 ppm
I20260927 18:46:00.054044 8706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:00.062186 8706 webserver.cc:460] Webserver started at http://127.31.26.254:33249/ using document root <none> and password file <none>
I20260927 18:46:00.062981 8706 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:00.063107 8706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:00.063462 8706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:00.066592 8706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "1a549ed435c44b0ebd0855d22fce12cc"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-rfd5"
server_key: "71e1e867e4bfc139ac300fe86a4d1a6c"
server_key_iv: "dfd84c3870f48fef0e3b4a1969fac59f"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.067437 8706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "1a549ed435c44b0ebd0855d22fce12cc"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-rfd5"
server_key: "71e1e867e4bfc139ac300fe86a4d1a6c"
server_key_iv: "dfd84c3870f48fef0e3b4a1969fac59f"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.072961 8706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 18:46:00.076754 8720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.078388 8706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:00.078622 8706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "1a549ed435c44b0ebd0855d22fce12cc"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-rfd5"
server_key: "71e1e867e4bfc139ac300fe86a4d1a6c"
server_key_iv: "dfd84c3870f48fef0e3b4a1969fac59f"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.078864 8706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:00.099354 8706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:00.100250 8706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:00.100541 8706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:00.122484 8706 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:35839
I20260927 18:46:00.122506 8771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:35839 every 8 connection(s)
I20260927 18:46:00.123968 8706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:00.127929 8772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:00.133697 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8706
I20260927 18:46:00.133898 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:46:00.134577 31851 external_mini_cluster.cc:1428] Setting key 5bcbc24dce95eb13861a25c240673046
I20260927 18:46:00.139055 8772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc: Bootstrap starting.
I20260927 18:46:00.142251 8772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:00.143283 8772 log.cc:824] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:00.146509 8772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc: No bootstrap required, opened a new log
I20260927 18:46:00.149003 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.136768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.31.26.254:35839)
0927 18:46:00.137820 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.137832 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.137873 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:46:00.139595 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.139622 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.140096 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.140518 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.141128 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.142396 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.143906 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.144730 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.144913 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.146108 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.146119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.146139 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:00.146874 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.147006 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:00.147042 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.147175 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.147366 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.147701 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":362,"threads_started":1}
I20260927 18:46:00.155279 8772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a549ed435c44b0ebd0855d22fce12cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35839 } }
I20260927 18:46:00.155634 8772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:00.155709 8772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a549ed435c44b0ebd0855d22fce12cc, State: Initialized, Role: FOLLOWER
I20260927 18:46:00.156208 8772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a549ed435c44b0ebd0855d22fce12cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35839 } }
I20260927 18:46:00.156395 8772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:00.156500 8772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:00.156626 8772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:00.159066 8772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a549ed435c44b0ebd0855d22fce12cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35839 } }
I20260927 18:46:00.159538 8772 leader_election.cc:302] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a549ed435c44b0ebd0855d22fce12cc; no voters:
I20260927 18:46:00.160066 8772 leader_election.cc:288] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:00.160230 8777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:00.160965 8777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [term 1 LEADER]: Becoming Leader. State: Replica: 1a549ed435c44b0ebd0855d22fce12cc, State: Running, Role: LEADER
I20260927 18:46:00.161645 8777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a549ed435c44b0ebd0855d22fce12cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35839 } }
I20260927 18:46:00.162180 8772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:00.164331 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a549ed435c44b0ebd0855d22fce12cc. Latest consensus state: current_term: 1 leader_uuid: "1a549ed435c44b0ebd0855d22fce12cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a549ed435c44b0ebd0855d22fce12cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35839 } } }
I20260927 18:46:00.164675 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:00.165578 8782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:00.165436 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a549ed435c44b0ebd0855d22fce12cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a549ed435c44b0ebd0855d22fce12cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35839 } } }
I20260927 18:46:00.165745 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:00.171036 8782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:00.179338 8782 catalog_manager.cc:1347] Generated new cluster ID: c46fac8a8ddd4ef8ae4e0c87803af46f
I20260927 18:46:00.179478 8782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:00.203055 8782 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:00.204362 8782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:00.210129 8782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a549ed435c44b0ebd0855d22fce12cc: Generated new TSK 0
I20260927 18:46:00.210963 8782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:00.222764 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35839
--builtin_ntp_servers=127.31.26.212:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:46:00.609298 8796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:00.609608 8796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:00.609732 8796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:00.617190 8796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:00.617359 8796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:00.617511 8796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:46:00.626791 8796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:00.628197 8796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:00.629987 8796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:00.643718 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.643782 8802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:00.644145 8801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:00.645133 8796 server_base.cc:1034] running on GCE node
I20260927 18:46:00.645816 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:00.646786 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:00.648036 8796 hybrid_clock.cc:648] HybridClock initialized: now 1790534760647958 us; error 53 us; skew 500 ppm
I20260927 18:46:00.648393 8796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:00.650887 8796 webserver.cc:460] Webserver started at http://127.31.26.193:34961/ using document root <none> and password file <none>
I20260927 18:46:00.651602 8796 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:00.651690 8796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:00.651969 8796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:00.655012 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "cd61235739ec42c3825240aba2c68818"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-rfd5"
server_key: "8832cfd4fcbda087a4cedd8b247103d8"
server_key_iv: "799b1fd620455877ade1503a065f0e0e"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.655843 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "cd61235739ec42c3825240aba2c68818"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-rfd5"
server_key: "8832cfd4fcbda087a4cedd8b247103d8"
server_key_iv: "799b1fd620455877ade1503a065f0e0e"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.661430 8796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 18:46:00.665474 8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.667012 8796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:00.667176 8796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "cd61235739ec42c3825240aba2c68818"
format_stamp: "Formatted at 2026-09-27 18:46:00 on dist-test-slave-rfd5"
server_key: "8832cfd4fcbda087a4cedd8b247103d8"
server_key_iv: "799b1fd620455877ade1503a065f0e0e"
server_key_version: "encryptionkey@0"
I20260927 18:46:00.667364 8796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:00.685854 8796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:00.686745 8796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:00.687057 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:00.688066 8796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:00.689664 8796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:00.689755 8796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.689903 8796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:00.689994 8796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:00.723644 8796 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:40901
I20260927 18:46:00.723688 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:40901 every 8 connection(s)
I20260927 18:46:00.724968 8796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:00.729732 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8796
I20260927 18:46:00.729912 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:00.730525 31851 external_mini_cluster.cc:1428] Setting key a218e5fed6978aad8ee4f7a10e5b29f2
I20260927 18:46:00.733402 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35839
--builtin_ntp_servers=127.31.26.212:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:46:00.738041 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.727613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:33777 (local address 127.31.26.254:35839)
0927 18:46:00.728081 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:00.728086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:00.729232 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 18:46:00.731123 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:00.731128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:00.731201 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:00.731341 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:00.732810 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.733664 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.735265 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.735655 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.735786 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.737206 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:00.737211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:00.737215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:00.737237 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:00.737276 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:00.737288 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:00.737384 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:00.737598 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:46:00.737693 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260927 18:46:00.739069 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:00.728088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35839 (local address 127.31.26.193:33777)
0927 18:46:00.729027 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:00.729069 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:00.730852 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:00.731658 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:00.731669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:00.731790 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:00.732636 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.732658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.733802 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.733807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.735101 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:00.735118 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:00.735894 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:00.735901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:00.736135 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:00.736858 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:00.736861 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:00.736903 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:00.736982 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:00.737389 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:00.737402 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:00.737411 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:46:00.737592 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 18:46:00.738002 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":289,"threads_started":1}
I20260927 18:46:00.740619 8923 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35839
I20260927 18:46:00.740932 8923 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:00.741990 8923 heartbeater.cc:511] Master 127.31.26.254:35839 requested a full tablet report, sending...
I20260927 18:46:00.744279 8737 ts_manager.cc:194] Registered new tserver with Master: cd61235739ec42c3825240aba2c68818 (127.31.26.193:40901)
I20260927 18:46:00.746501 8737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.193:33777
W20260927 18:46:01.064785 8927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:01.065076 8927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:01.065186 8927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:01.072546 8927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:01.072664 8927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:01.072773 8927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:46:01.082779 8927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:01.084203 8927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:01.086203 8927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:01.100303 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.100494 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.100322 8932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.101150 8927 server_base.cc:1034] running on GCE node
I20260927 18:46:01.101850 8927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:01.102862 8927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:01.104108 8927 hybrid_clock.cc:648] HybridClock initialized: now 1790534761104063 us; error 43 us; skew 500 ppm
I20260927 18:46:01.104447 8927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:01.107090 8927 webserver.cc:460] Webserver started at http://127.31.26.194:36971/ using document root <none> and password file <none>
I20260927 18:46:01.107841 8927 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:01.107966 8927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:01.108290 8927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:01.111377 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "0675945ec16d448fb343d8891076f262"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-rfd5"
server_key: "df075b058118f55d38a395763454804a"
server_key_iv: "a83358d6c959666cb64a449d590ce768"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.112224 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "0675945ec16d448fb343d8891076f262"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-rfd5"
server_key: "df075b058118f55d38a395763454804a"
server_key_iv: "a83358d6c959666cb64a449d590ce768"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.117758 8927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:46:01.121326 8941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.123013 8927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:01.123207 8927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "0675945ec16d448fb343d8891076f262"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-rfd5"
server_key: "df075b058118f55d38a395763454804a"
server_key_iv: "a83358d6c959666cb64a449d590ce768"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.123452 8927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:01.143436 8927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:01.144532 8927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:01.144881 8927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:01.145928 8927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:01.147399 8927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:01.147483 8927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.147569 8927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:01.147624 8927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.180998 8927 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:37155
I20260927 18:46:01.181022 9053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:37155 every 8 connection(s)
I20260927 18:46:01.182384 8927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:01.188398 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 8927
I20260927 18:46:01.188540 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:01.188987 31851 external_mini_cluster.cc:1428] Setting key f52d712fab32df771289bf5c1e7eaa60
I20260927 18:46:01.191867 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35839
--builtin_ntp_servers=127.31.26.212:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:46:01.195690 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.185208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:33931 (local address 127.31.26.254:35839)
0927 18:46:01.185407 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.185416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.186995 (+ 1579us) server_negotiation.cc:407] Connection header received
0927 18:46:01.188785 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.188790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.188884 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.188994 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.190296 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.191139 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.192826 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.193320 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.193455 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.194900 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.194905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.194908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.194926 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.194956 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.194968 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.195058 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.195287 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.195386 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:46:01.196802 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.185883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35839 (local address 127.31.26.194:33931)
0927 18:46:01.186826 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.186874 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:01.188417 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:01.189272 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:01.189281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:01.189378 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:01.190130 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.190149 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.191296 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.191301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.192681 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.192693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.193548 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.193554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.193809 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.194556 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:01.194558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:01.194611 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:01.194694 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:01.195067 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:01.195080 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:01.195091 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:46:01.195278 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 18:46:01.195710 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
I20260927 18:46:01.198298 9054 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35839
I20260927 18:46:01.198607 9054 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:01.199527 9054 heartbeater.cc:511] Master 127.31.26.254:35839 requested a full tablet report, sending...
I20260927 18:46:01.201112 8737 ts_manager.cc:194] Registered new tserver with Master: 0675945ec16d448fb343d8891076f262 (127.31.26.194:37155)
I20260927 18:46:01.202353 8737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.194:33931
W20260927 18:46:01.529206 9057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:01.529699 9057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:01.529870 9057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:01.537680 9057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:01.537801 9057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:01.537966 9057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:46:01.547299 9057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:01.548653 9057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:01.550484 9057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:01.563632 9062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.563899 9063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:01.563632 9065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:01.564702 9057 server_base.cc:1034] running on GCE node
I20260927 18:46:01.565445 9057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:01.566427 9057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:01.567652 9057 hybrid_clock.cc:648] HybridClock initialized: now 1790534761567600 us; error 45 us; skew 500 ppm
I20260927 18:46:01.568010 9057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:01.570521 9057 webserver.cc:460] Webserver started at http://127.31.26.195:39377/ using document root <none> and password file <none>
I20260927 18:46:01.571336 9057 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:01.571463 9057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:01.571794 9057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:01.574815 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "7f6501f36d694e49b1248601c182e6a9"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-rfd5"
server_key: "e01cf4ea0957f4c57e70eea4b8787815"
server_key_iv: "2146a078c820a45683e2370c0e24fab5"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.575635 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "7f6501f36d694e49b1248601c182e6a9"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-rfd5"
server_key: "e01cf4ea0957f4c57e70eea4b8787815"
server_key_iv: "2146a078c820a45683e2370c0e24fab5"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.580865 9057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:01.584662 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.586105 9057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:01.586290 9057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "7f6501f36d694e49b1248601c182e6a9"
format_stamp: "Formatted at 2026-09-27 18:46:01 on dist-test-slave-rfd5"
server_key: "e01cf4ea0957f4c57e70eea4b8787815"
server_key_iv: "2146a078c820a45683e2370c0e24fab5"
server_key_version: "encryptionkey@0"
I20260927 18:46:01.586516 9057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:01.610968 9057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:01.611882 9057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:01.612277 9057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:01.613385 9057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:01.614960 9057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:01.615095 9057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.615218 9057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:01.615303 9057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:01.648742 9057 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35277
I20260927 18:46:01.648810 9183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35277 every 8 connection(s)
I20260927 18:46:01.650137 9057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:01.656798 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9057
I20260927 18:46:01.656981 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:01.657439 31851 external_mini_cluster.cc:1428] Setting key ca36dec0237ddeef545ac48e9252523f
I20260927 18:46:01.663584 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.652948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:41903 (local address 127.31.26.254:35839)
0927 18:46:01.653168 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.653173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.654671 (+ 1498us) server_negotiation.cc:407] Connection header received
0927 18:46:01.656484 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.656489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.656592 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.656699 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.657992 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.658790 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.660894 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.661343 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.661453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.662856 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.662861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.662864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.662882 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.662929 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.662958 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.663044 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.663253 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.663332 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:46:01.664739 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.653535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35839 (local address 127.31.26.195:41903)
0927 18:46:01.654511 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:01.654546 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:01.656177 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:01.656978 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:01.656987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:01.657157 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:01.657797 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.657816 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.658947 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.658951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.660765 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:01.660777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.661570 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.661575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:01.661798 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.662520 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:01.662522 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:01.662566 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:01.662674 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:01.663068 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:01.663077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:01.663084 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:46:01.663236 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:46:01.663564 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
I20260927 18:46:01.666244 9184 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35839
I20260927 18:46:01.666528 9184 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:01.667281 9184 heartbeater.cc:511] Master 127.31.26.254:35839 requested a full tablet report, sending...
I20260927 18:46:01.668829 8737 ts_manager.cc:194] Registered new tserver with Master: 7f6501f36d694e49b1248601c182e6a9 (127.31.26.195:35277)
I20260927 18:46:01.669716 8737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.195:41903
I20260927 18:46:01.672991 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:46:01.679402 8925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:01.677829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.31.26.254:35839)
0927 18:46:01.677990 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.677996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.678529 (+ 533us) server_negotiation.cc:407] Connection header received
0927 18:46:01.678685 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.678689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.678762 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.678857 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.679220 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38468: BlockingRecv error: recv got EOF from 127.0.0.1:38468 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:46:01.680091 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8796
I20260927 18:46:01.688342 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.681201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.31.26.193:40901)
0927 18:46:01.681746 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.681754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.681782 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:46:01.681837 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.681843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.682083 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.682236 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.682590 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.683659 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.684956 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.685589 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.685687 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.686648 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.686656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.686670 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.687291 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.687338 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.687358 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.687450 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.687712 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.687978 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":277,"threads_started":1}
I20260927 18:46:01.689738 8918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:01.770704 8923 heartbeater.cc:503] Master 127.31.26.254:35839 was elected leader, sending a full tablet report...
I20260927 18:46:01.771822 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8796
W20260927 18:46:01.787777 8723 connection.cc:535] server connection from 127.31.26.193:33777 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:46:01.788245 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8927
I20260927 18:46:01.796443 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.789196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.31.26.194:37155)
0927 18:46:01.789763 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.789771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.789799 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:46:01.789863 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.789870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.790138 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.790287 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.790688 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.791788 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.793330 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.793904 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.794004 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.795109 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.795116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.795129 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.795725 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.795774 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.795792 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.795879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.796030 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.796150 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260927 18:46:01.797713 9049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:01.878822 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8927
I20260927 18:46:01.895056 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9057
I20260927 18:46:01.903182 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:01.896072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.31.26.195:35277)
0927 18:46:01.896730 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:01.896737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:01.896768 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:46:01.896825 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:01.896831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:01.897057 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:01.897241 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:01.897603 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.898607 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.899817 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:01.900368 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:01.900459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:01.901650 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:01.901659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:01.901671 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:01.902286 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:01.902333 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:01.902352 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:01.902463 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:01.902598 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:46:01.902703 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":355,"threads_started":1}
I20260927 18:46:01.904546 9179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:01.985751 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9057
I20260927 18:46:02.001806 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 8706
I20260927 18:46:02.003065 8767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:02.103735 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 8706
2026-09-27T18:46:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2496 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T18:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:02Z Disabled control of system clock
I20260927 18:46:02.176728 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:38149
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:38149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:46:02.511565 9203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:02.511901 9203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:02.512001 9203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:02.519629 9203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:02.519761 9203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:02.519801 9203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:02.519928 9203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:02.520015 9203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:02.529567 9203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:38149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:38149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:02.530946 9203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:02.532835 9203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:02.545244 9211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:02.545274 9208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:02.545190 9209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:02.546126 9203 server_base.cc:1034] running on GCE node
I20260927 18:46:02.547020 9203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:02.548513 9203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:02.549739 9203 hybrid_clock.cc:648] HybridClock initialized: now 1790534762549690 us; error 47 us; skew 500 ppm
I20260927 18:46:02.550112 9203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:02.552556 9203 webserver.cc:460] Webserver started at http://127.31.26.254:44447/ using document root <none> and password file <none>
I20260927 18:46:02.553375 9203 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:02.553504 9203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:02.553844 9203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:02.558199 9203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "be2cbf78e97d444ab68818e313fc99ef"
format_stamp: "Formatted at 2026-09-27 18:46:02 on dist-test-slave-rfd5"
server_key: "2eaabb626df89559b217684dfe074d2a"
server_key_iv: "3aaff288cc03fad4709abb12a43cba64"
server_key_version: "encryptionkey@0"
I20260927 18:46:02.559108 9203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "be2cbf78e97d444ab68818e313fc99ef"
format_stamp: "Formatted at 2026-09-27 18:46:02 on dist-test-slave-rfd5"
server_key: "2eaabb626df89559b217684dfe074d2a"
server_key_iv: "3aaff288cc03fad4709abb12a43cba64"
server_key_version: "encryptionkey@0"
I20260927 18:46:02.564706 9203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:02.568627 9217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:02.570232 9203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:02.570436 9203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "be2cbf78e97d444ab68818e313fc99ef"
format_stamp: "Formatted at 2026-09-27 18:46:02 on dist-test-slave-rfd5"
server_key: "2eaabb626df89559b217684dfe074d2a"
server_key_iv: "3aaff288cc03fad4709abb12a43cba64"
server_key_version: "encryptionkey@0"
I20260927 18:46:02.570676 9203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:02.576690 9203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:02.577613 9203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:02.577919 9203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:02.596669 9203 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:38149
I20260927 18:46:02.596705 9268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:38149 every 8 connection(s)
I20260927 18:46:02.598182 9203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:02.602394 9269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:02.608565 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9203
I20260927 18:46:02.608819 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:46:02.609678 31851 external_mini_cluster.cc:1428] Setting key 0480914847d2bf73983d4267d42d6700
I20260927 18:46:02.614897 9269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef: Bootstrap starting.
I20260927 18:46:02.618881 9269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:02.619454 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:02.612152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.31.26.254:38149)
0927 18:46:02.613382 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:02.613395 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:02.613442 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:46:02.615307 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:02.615342 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:02.615823 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:02.616207 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:02.617514 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:02.617522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:02.617537 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:02.617554 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:02.617631 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:02.617649 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:02.617850 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:02.617988 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:46:02.618317 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":428,"threads_started":1}
I20260927 18:46:02.620286 9269 log.cc:824] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:02.623732 9269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef: No bootstrap required, opened a new log
I20260927 18:46:02.631280 9269 raft_consensus.cc:348] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2cbf78e97d444ab68818e313fc99ef" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 38149 } }
I20260927 18:46:02.631695 9269 raft_consensus.cc:374] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:02.631821 9269 raft_consensus.cc:729] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2cbf78e97d444ab68818e313fc99ef, State: Initialized, Role: FOLLOWER
I20260927 18:46:02.632584 9269 consensus_queue.cc:260] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2cbf78e97d444ab68818e313fc99ef" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 38149 } }
I20260927 18:46:02.632872 9269 raft_consensus.cc:388] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:02.633009 9269 raft_consensus.cc:482] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:02.633257 9269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:02.635490 9269 raft_consensus.cc:504] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2cbf78e97d444ab68818e313fc99ef" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 38149 } }
I20260927 18:46:02.636065 9269 leader_election.cc:302] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be2cbf78e97d444ab68818e313fc99ef; no voters:
I20260927 18:46:02.636688 9269 leader_election.cc:288] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:02.636838 9274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:02.637635 9274 raft_consensus.cc:686] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [term 1 LEADER]: Becoming Leader. State: Replica: be2cbf78e97d444ab68818e313fc99ef, State: Running, Role: LEADER
I20260927 18:46:02.638336 9274 consensus_queue.cc:237] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2cbf78e97d444ab68818e313fc99ef" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 38149 } }
I20260927 18:46:02.639217 9269 sys_catalog.cc:564] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:02.641350 9275 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be2cbf78e97d444ab68818e313fc99ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2cbf78e97d444ab68818e313fc99ef" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 38149 } } }
I20260927 18:46:02.641578 9276 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader be2cbf78e97d444ab68818e313fc99ef. Latest consensus state: current_term: 1 leader_uuid: "be2cbf78e97d444ab68818e313fc99ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2cbf78e97d444ab68818e313fc99ef" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 38149 } } }
I20260927 18:46:02.641793 9275 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:02.641867 9276 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:02.644351 9282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:02.649659 9282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:02.658448 9282 catalog_manager.cc:1347] Generated new cluster ID: 5a6530fdecdd4231819ec38d685b8049
I20260927 18:46:02.658603 9282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:02.680797 9282 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:02.681358 9282 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:02.699524 9282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be2cbf78e97d444ab68818e313fc99ef: Generated new TSK 0
I20260927 18:46:02.700704 9282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:02.715788 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:38149
--builtin_ntp_servers=127.31.26.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:46:03.056246 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.056598 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.056756 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.064157 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.064302 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.064486 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:46:03.073787 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:38149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:03.075246 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.077037 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.089845 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.090015 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.090056 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.091005 9293 server_base.cc:1034] running on GCE node
I20260927 18:46:03.091661 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.092677 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.093925 9293 hybrid_clock.cc:648] HybridClock initialized: now 1790534763093879 us; error 55 us; skew 500 ppm
I20260927 18:46:03.094265 9293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:03.096626 9293 webserver.cc:460] Webserver started at http://127.31.26.193:41297/ using document root <none> and password file <none>
I20260927 18:46:03.097472 9293 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.097600 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.097921 9293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.100745 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "28ef2193d05f44ce896face5986edaea"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "f26cfb8eed1736b8e76f185bea37dda7"
server_key_iv: "208fd4e1f171b75adc6c1cf0aca17e38"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.101564 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "28ef2193d05f44ce896face5986edaea"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "f26cfb8eed1736b8e76f185bea37dda7"
server_key_iv: "208fd4e1f171b75adc6c1cf0aca17e38"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.107002 9293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:03.110450 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.111792 9293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:03.111991 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "28ef2193d05f44ce896face5986edaea"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "f26cfb8eed1736b8e76f185bea37dda7"
server_key_iv: "208fd4e1f171b75adc6c1cf0aca17e38"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.112217 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:03.117985 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:03.118724 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:03.118995 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:03.119904 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:03.121443 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:03.121531 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.121614 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:03.121668 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.154852 9293 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:39007
I20260927 18:46:03.154893 9419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:39007 every 8 connection(s)
I20260927 18:46:03.156229 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:03.160334 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9293
I20260927 18:46:03.160574 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:03.161290 31851 external_mini_cluster.cc:1428] Setting key d846d1a4c73d1c92cd453271c01df78d
I20260927 18:46:03.164342 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:38149
--builtin_ntp_servers=127.31.26.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:46:03.165642 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.159083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:33081 (local address 127.31.26.254:38149)
0927 18:46:03.159565 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:03.159570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:03.161060 (+ 1490us) server_negotiation.cc:407] Connection header received
0927 18:46:03.163181 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:03.163185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:03.163231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:03.163353 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:03.164720 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:03.164728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:03.164735 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:03.164757 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:03.164808 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:03.164827 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:03.164961 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:03.165199 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:46:03.165263 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260927 18:46:03.166440 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.159739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:38149 (local address 127.31.26.193:33081)
0927 18:46:03.160861 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:03.160908 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:03.162852 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:03.163568 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:03.163577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:03.163675 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:03.164380 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:03.164383 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:03.164427 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:03.164545 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:03.164928 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:03.164935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:03.164941 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:03.165249 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 18:46:03.165459 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":366,"threads_started":1}
I20260927 18:46:03.167917 9420 heartbeater.cc:347] Connected to a master server at 127.31.26.254:38149
I20260927 18:46:03.168237 9420 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:03.168936 9420 heartbeater.cc:511] Master 127.31.26.254:38149 requested a full tablet report, sending...
I20260927 18:46:03.171020 9234 ts_manager.cc:194] Registered new tserver with Master: 28ef2193d05f44ce896face5986edaea (127.31.26.193:39007)
W20260927 18:46:03.498510 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.498803 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.498950 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.506418 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.506541 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.506654 9424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:46:03.516031 9424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:38149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:03.517542 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.519405 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.532925 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.532974 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.533263 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.533875 9424 server_base.cc:1034] running on GCE node
I20260927 18:46:03.534523 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.535447 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.536649 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790534763536601 us; error 40 us; skew 500 ppm
I20260927 18:46:03.536999 9424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:03.539644 9424 webserver.cc:460] Webserver started at http://127.31.26.194:41099/ using document root <none> and password file <none>
I20260927 18:46:03.540450 9424 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.540566 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.540963 9424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.544030 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "d5af5f2105454c8b8adeb8c485e18031"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "d73a10e16ab225f4f2722fea42814989"
server_key_iv: "260f226c31ed699524bca425139758b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.544911 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "d5af5f2105454c8b8adeb8c485e18031"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "d73a10e16ab225f4f2722fea42814989"
server_key_iv: "260f226c31ed699524bca425139758b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.550339 9424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:46:03.554028 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.555492 9424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 18:46:03.555666 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "d5af5f2105454c8b8adeb8c485e18031"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "d73a10e16ab225f4f2722fea42814989"
server_key_iv: "260f226c31ed699524bca425139758b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.555894 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:03.561781 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:03.562628 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:03.562891 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:03.563795 9424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:03.565253 9424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:03.565363 9424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.565487 9424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:03.565577 9424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.598821 9424 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:36179
I20260927 18:46:03.598862 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:36179 every 8 connection(s)
I20260927 18:46:03.600251 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:03.608682 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9424
I20260927 18:46:03.608891 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:03.609519 31851 external_mini_cluster.cc:1428] Setting key fd103acb40980fded85805c068ab63a3
I20260927 18:46:03.609927 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.603003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:33195 (local address 127.31.26.254:38149)
0927 18:46:03.603220 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:03.603226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:03.604917 (+ 1691us) server_negotiation.cc:407] Connection header received
0927 18:46:03.607328 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:03.607333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:03.607409 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:03.607518 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:03.609164 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:03.609173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:03.609178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:03.609198 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:03.609256 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:03.609269 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:03.609403 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:03.609607 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 18:46:03.609674 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:46:03.611044 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:03.603658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:38149 (local address 127.31.26.194:33195)
0927 18:46:03.604734 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:03.604786 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:03.607004 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:03.607780 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:03.607791 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:03.607924 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:03.608721 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:03.608724 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:03.608763 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:03.608909 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:03.609361 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:03.609370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:03.609379 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:46:03.609677 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 18:46:03.609944 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":304,"threads_started":1}
I20260927 18:46:03.612296 9551 heartbeater.cc:347] Connected to a master server at 127.31.26.254:38149
I20260927 18:46:03.612612 9551 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:03.612690 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:38149
--builtin_ntp_servers=127.31.26.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:46:03.613327 9551 heartbeater.cc:511] Master 127.31.26.254:38149 requested a full tablet report, sending...
I20260927 18:46:03.614722 9234 ts_manager.cc:194] Registered new tserver with Master: d5af5f2105454c8b8adeb8c485e18031 (127.31.26.194:36179)
W20260927 18:46:03.941391 9554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:03.941742 9554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:03.941903 9554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:03.949461 9554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:03.949622 9554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:03.949798 9554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:46:03.959100 9554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:38149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:03.960533 9554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:03.962440 9554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:03.977061 9559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.977258 9560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:03.977269 9562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:03.978099 9554 server_base.cc:1034] running on GCE node
I20260927 18:46:03.978724 9554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:03.979701 9554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:03.980927 9554 hybrid_clock.cc:648] HybridClock initialized: now 1790534763980870 us; error 57 us; skew 500 ppm
I20260927 18:46:03.981304 9554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:46:03.983695 9554 webserver.cc:460] Webserver started at http://127.31.26.195:41113/ using document root <none> and password file <none>
I20260927 18:46:03.984421 9554 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:03.984555 9554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:03.984864 9554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:03.987810 9554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "799878f2d5494aa0b53a2092b53a9392"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "108ce0a3fe51827f7399b0a71bc40764"
server_key_iv: "e2c965669226ce36a885991fde3e5874"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.988637 9554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "799878f2d5494aa0b53a2092b53a9392"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "108ce0a3fe51827f7399b0a71bc40764"
server_key_iv: "e2c965669226ce36a885991fde3e5874"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.993994 9554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 18:46:03.997474 9568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:03.998924 9554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:46:03.999132 9554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "799878f2d5494aa0b53a2092b53a9392"
format_stamp: "Formatted at 2026-09-27 18:46:03 on dist-test-slave-rfd5"
server_key: "108ce0a3fe51827f7399b0a71bc40764"
server_key_iv: "e2c965669226ce36a885991fde3e5874"
server_key_version: "encryptionkey@0"
I20260927 18:46:03.999368 9554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:04.005266 9554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:04.006075 9554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:04.006345 9554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:04.007267 9554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:04.008665 9554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:04.008782 9554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.008912 9554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:04.008998 9554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:04.039997 9554 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:39317
I20260927 18:46:04.040081 9680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:39317 every 8 connection(s)
I20260927 18:46:04.041368 9554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:04.046007 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9554
I20260927 18:46:04.046218 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:04.046779 31851 external_mini_cluster.cc:1428] Setting key 3aa6ca89d47ba85559b39a8d31ee2d4e
I20260927 18:46:04.051062 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.044212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:47849 (local address 127.31.26.254:38149)
0927 18:46:04.044393 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.044398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.046140 (+ 1742us) server_negotiation.cc:407] Connection header received
0927 18:46:04.048517 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.048521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.048626 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.048737 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.050402 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.050407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.050410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:04.050429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.050466 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:04.050478 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.050602 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.050772 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.050830 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:46:04.052178 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.044814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:38149 (local address 127.31.26.195:47849)
0927 18:46:04.045947 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:04.046002 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:04.048173 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:04.049143 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:04.049157 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:04.049278 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:04.050072 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:46:04.050075 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:04.050118 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:46:04.050210 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:04.050582 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:04.050589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:04.050595 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:04.050820 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 18:46:04.051086 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":330,"threads_started":1}
I20260927 18:46:04.053735 9681 heartbeater.cc:347] Connected to a master server at 127.31.26.254:38149
I20260927 18:46:04.054113 9681 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:04.054724 9681 heartbeater.cc:511] Master 127.31.26.254:38149 requested a full tablet report, sending...
I20260927 18:46:04.056234 9234 ts_manager.cc:194] Registered new tserver with Master: 799878f2d5494aa0b53a2092b53a9392 (127.31.26.195:39317)
I20260927 18:46:04.062774 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:46:04.069137 9422 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:04.067538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.31.26.254:38149)
0927 18:46:04.067709 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.067714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.068190 (+ 476us) server_negotiation.cc:407] Connection header received
0927 18:46:04.068333 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.068337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.068388 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.068487 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.068860 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44682: BlockingRecv error: recv got EOF from 127.0.0.1:44682 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:46:04.071327 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9293
I20260927 18:46:04.075181 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.072361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.31.26.193:39007)
0927 18:46:04.072934 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.072941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.072968 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:46:04.073025 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.073031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.073347 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.073477 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.074407 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.074414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.074426 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:04.074444 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.074487 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:04.074502 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.074680 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.074739 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.074849 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":314,"threads_started":1}
I20260927 18:46:04.076434 9415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:04.165736 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9293
I20260927 18:46:04.182080 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9424
I20260927 18:46:04.185883 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.183018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.31.26.194:36179)
0927 18:46:04.183542 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.183550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.183577 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:46:04.183637 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.183643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.183908 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.184106 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.185150 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.185159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.185179 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:04.185198 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.185239 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:04.185254 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.185394 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.185500 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.185603 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":260,"threads_started":1}
I20260927 18:46:04.187119 9546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:04.269318 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9424
I20260927 18:46:04.287801 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9554
I20260927 18:46:04.291534 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:04.288802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.31.26.195:39317)
0927 18:46:04.289458 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:04.289465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:04.289496 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:46:04.289563 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:04.289569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:04.289787 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:04.289912 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:04.290829 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:04.290836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:04.290847 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:04.290858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:04.290895 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:04.290908 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:04.291050 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:04.291100 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 18:46:04.291210 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":369,"threads_started":1}
I20260927 18:46:04.292900 9676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:04.375792 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9554
I20260927 18:46:04.391736 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9203
I20260927 18:46:04.392980 9264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:04.496655 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9203
2026-09-27T18:46:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2390 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:04 dist-test-slave-rfd5 krb5kdc[9699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:04 dist-test-slave-rfd5 krb5kdc[9699](info): set up 2 sockets
Sep 27 18:46:04 dist-test-slave-rfd5 krb5kdc[9699](info): commencing operation
krb5kdc: starting...
W20260927 18:46:05.610123 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:46:05 dist-test-slave-rfd5 krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:46:05.844907 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41723
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:06.183760 9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:06.184082 9712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:06.184237 9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:06.191781 9712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:06.191915 9712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:06.191999 9712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:06.192090 9712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:06.192180 9712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:06.202651 9712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:06.204090 9712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:06.205497 9712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:06.217976 9717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.217962 9718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.218015 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:06.218755 9712 server_base.cc:1034] running on GCE node
I20260927 18:46:06.219785 9712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:06.221484 9712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:06.222749 9712 hybrid_clock.cc:648] HybridClock initialized: now 1790534766222707 us; error 54 us; skew 500 ppm
I20260927 18:46:06.223137 9712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:06 dist-test-slave-rfd5 krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:06.230532 9712 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:46:06.232091 9712 webserver.cc:460] Webserver started at http://127.31.26.254:36685/ using document root <none> and password file <none>
I20260927 18:46:06.232831 9712 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:06.232986 9712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:06.233345 9712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:06.236696 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "3c2ef4d530c5474dacc8f8836dae3aed"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-rfd5"
server_key: "f2793f8595e3cfcc53552d6e3a452fb8"
server_key_iv: "fd5f8adbf0366c9b6263e85bfbb0eee1"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.237561 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "3c2ef4d530c5474dacc8f8836dae3aed"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-rfd5"
server_key: "f2793f8595e3cfcc53552d6e3a452fb8"
server_key_iv: "fd5f8adbf0366c9b6263e85bfbb0eee1"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.242832 9712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:06.246711 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.248503 9712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:06.248747 9712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "3c2ef4d530c5474dacc8f8836dae3aed"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-rfd5"
server_key: "f2793f8595e3cfcc53552d6e3a452fb8"
server_key_iv: "fd5f8adbf0366c9b6263e85bfbb0eee1"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.249053 9712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:06.268482 9712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:06.272944 9712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:06.273296 9712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:06.292346 9712 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41723
I20260927 18:46:06.292369 9778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41723 every 8 connection(s)
I20260927 18:46:06.293857 9712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:06.297731 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:06.298329 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9712
I20260927 18:46:06.298574 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:46:06.299218 31851 external_mini_cluster.cc:1428] Setting key d85315afbfc9e5e6797f0744106f0592
I20260927 18:46:06.312165 9779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed: Bootstrap starting.
Sep 27 18:46:06 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:06.315436 9779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:06.316862 9779 log.cc:824] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:06.320519 9779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed: No bootstrap required, opened a new log
I20260927 18:46:06.323588 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.301338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.31.26.254:41723)
0927 18:46:06.302173 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:06.302190 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:06.302253 (+ 63us) server_negotiation.cc:407] Connection header received
0927 18:46:06.304444 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:06.304474 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:06.305039 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:06.305530 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:06.306829 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.308852 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.310210 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.311275 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.311454 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.315449 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:06.315464 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:06.315486 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:06.315553 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:06.319394 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.320122 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.320131 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.320140 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.320337 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.320751 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.320755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.320757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.321357 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:06.321686 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:06.322152 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 18:46:06.322577 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":353,"threads_started":1}
I20260927 18:46:06.328715 9779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41723 } }
I20260927 18:46:06.329180 9779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:06.329300 9779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2ef4d530c5474dacc8f8836dae3aed, State: Initialized, Role: FOLLOWER
I20260927 18:46:06.329864 9779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41723 } }
I20260927 18:46:06.330108 9779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:06.330255 9779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:06.330425 9779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:06.332938 9779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41723 } }
I20260927 18:46:06.333563 9779 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2ef4d530c5474dacc8f8836dae3aed; no voters:
I20260927 18:46:06.334192 9779 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:06.334331 9784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:06.335062 9784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [term 1 LEADER]: Becoming Leader. State: Replica: 3c2ef4d530c5474dacc8f8836dae3aed, State: Running, Role: LEADER
I20260927 18:46:06.335691 9784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41723 } }
I20260927 18:46:06.336484 9779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:06.339736 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2ef4d530c5474dacc8f8836dae3aed. Latest consensus state: current_term: 1 leader_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41723 } } }
I20260927 18:46:06.340176 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:06.341189 9790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:06.340955 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2ef4d530c5474dacc8f8836dae3aed" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41723 } } }
I20260927 18:46:06.341514 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:06.348575 9790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:06.357841 9790 catalog_manager.cc:1347] Generated new cluster ID: fe412ce58fb34103b7ccd061480c7bce
I20260927 18:46:06.357977 9790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:06.365556 9790 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:06.367443 9790 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:06.381772 9790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed: Generated new TSK 0
I20260927 18:46:06.382730 9790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:46:06.452126 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41723
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:06.793565 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:06.793875 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:06.793973 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:06.801385 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:06.801510 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:06.801625 9805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:46:06.811604 9805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:06.813239 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:06.814625 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:06.828461 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.828563 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:06.828588 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:06.829444 9805 server_base.cc:1034] running on GCE node
I20260927 18:46:06.830065 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:06.831027 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:06.832232 9805 hybrid_clock.cc:648] HybridClock initialized: now 1790534766832162 us; error 64 us; skew 500 ppm
I20260927 18:46:06.832589 9805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:06 dist-test-slave-rfd5 krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:06.835978 9805 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:46:06.838079 9805 webserver.cc:460] Webserver started at http://127.31.26.193:41155/ using document root <none> and password file <none>
I20260927 18:46:06.838832 9805 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:06.838989 9805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:06.839301 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:06.842379 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "edd0802728e04f58965d1ef0774ddb83"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-rfd5"
server_key: "d7a57383a27e0f00f2e2012755be8e01"
server_key_iv: "684d946e78b8704171fc071f744554d3"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.843221 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "edd0802728e04f58965d1ef0774ddb83"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-rfd5"
server_key: "d7a57383a27e0f00f2e2012755be8e01"
server_key_iv: "684d946e78b8704171fc071f744554d3"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.849265 9805 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:46:06.852864 9820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.854710 9805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:06.854910 9805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "edd0802728e04f58965d1ef0774ddb83"
format_stamp: "Formatted at 2026-09-27 18:46:06 on dist-test-slave-rfd5"
server_key: "d7a57383a27e0f00f2e2012755be8e01"
server_key_iv: "684d946e78b8704171fc071f744554d3"
server_key_version: "encryptionkey@0"
I20260927 18:46:06.855163 9805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:06.871291 9805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:06.876113 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:06.876488 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:06.877678 9805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:06.879223 9805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:06.879333 9805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.879485 9805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:06.879561 9805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:06.913582 9805 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34037
I20260927 18:46:06.913637 9932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34037 every 8 connection(s)
I20260927 18:46:06.915045 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:06.916075 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9805
I20260927 18:46:06.916301 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:06.916975 31851 external_mini_cluster.cc:1428] Setting key fd8f59a98854252ad8c82b0d7f94a42b
Sep 27 18:46:06 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:06.936285 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.917889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:48173 (local address 127.31.26.254:41723)
0927 18:46:06.918336 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:06.918340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:06.919516 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 18:46:06.921376 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:06.921384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:06.921472 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:06.921596 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:06.923103 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.924281 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.926209 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.926616 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.926717 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.930412 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:06.930419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:06.930431 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:06.930467 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:06.933047 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.933878 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.933884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.933888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.934025 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:06.934484 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:06.934489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:06.934493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:06.934767 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:06.934927 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:06.935677 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 18:46:06.935907 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":247,"threads_started":1}
I20260927 18:46:06.937681 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:06.918456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41723 (local address 127.31.26.193:48173)
0927 18:46:06.919338 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:06.919389 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:06.921050 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:06.921808 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:06.921817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:06.922336 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:06.922926 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:06.922944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.924436 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.924440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:06.926029 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:06.926047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:06.926867 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:06.926877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:06.927151 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:06.927879 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:06.927904 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:06.930165 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:06.933240 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:06.933249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:06.933269 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:06.933738 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:06.934145 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:06.934150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:06.934154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:06.934361 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:06.934904 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:06.934912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:06.935288 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 18:46:06.935571 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 18:46:06.936154 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":379,"threads_started":1}
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:46:06.939468 9933 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41723
I20260927 18:46:06.939782 9933 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:06.940600 9933 heartbeater.cc:511] Master 127.31.26.254:41723 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:46:06.943001 9744 ts_manager.cc:194] Registered new tserver with Master: edd0802728e04f58965d1ef0774ddb83 (127.31.26.193:34037)
I20260927 18:46:06.944489 9744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:48173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:46:06.968302 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41723
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:07.313930 9939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:07.314260 9939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:07.314438 9939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:07.321991 9939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:07.322144 9939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:07.322312 9939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:46:07.332304 9939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:07.333698 9939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:07.335005 9939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:07.349962 9945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.349895 9944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.350692 9939 server_base.cc:1034] running on GCE node
W20260927 18:46:07.350690 9947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.351361 9939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:07.352350 9939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:07.353596 9939 hybrid_clock.cc:648] HybridClock initialized: now 1790534767353539 us; error 53 us; skew 500 ppm
I20260927 18:46:07.353965 9939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:07 dist-test-slave-rfd5 krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:07.357170 9939 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:46:07.358661 9939 webserver.cc:460] Webserver started at http://127.31.26.194:34949/ using document root <none> and password file <none>
I20260927 18:46:07.359422 9939 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:07.359566 9939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:07.359879 9939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:07.362883 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "1568e8ea1f7544409438c844722f42b1"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-rfd5"
server_key: "a99420b51865150404ab4c43cf6b9861"
server_key_iv: "268d2645ed65d4e5e7b5b4c3a0ad6dc2"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.363642 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "1568e8ea1f7544409438c844722f42b1"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-rfd5"
server_key: "a99420b51865150404ab4c43cf6b9861"
server_key_iv: "268d2645ed65d4e5e7b5b4c3a0ad6dc2"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.368676 9939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:46:07.372094 9954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.373468 9939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:07.373656 9939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "1568e8ea1f7544409438c844722f42b1"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-rfd5"
server_key: "a99420b51865150404ab4c43cf6b9861"
server_key_iv: "268d2645ed65d4e5e7b5b4c3a0ad6dc2"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.373847 9939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:07.420171 9939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:07.424412 9939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:07.424742 9939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:07.425735 9939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:07.427172 9939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:07.427274 9939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.427430 9939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:07.427503 9939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.461659 9939 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:39429
I20260927 18:46:07.461747 10066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:39429 every 8 connection(s)
I20260927 18:46:07.462949 9939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:07.464505 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 9939
I20260927 18:46:07.464718 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:07.465271 31851 external_mini_cluster.cc:1428] Setting key 83be0a9f324f3f2e2e816669e541b24b
Sep 27 18:46:07 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:07.485181 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.465772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:44181 (local address 127.31.26.254:41723)
0927 18:46:07.466265 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.466271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.467369 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 18:46:07.469175 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.469180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.469263 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.469385 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:07.470919 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.471662 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.473249 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.473981 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.474126 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.477850 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:07.477860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:07.477868 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:07.477914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:07.481668 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.482476 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.482485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.482491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.482664 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:07.483078 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:07.483086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:07.483092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:07.483493 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:07.483652 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:07.484366 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 18:46:07.484626 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":234,"threads_started":1}
I20260927 18:46:07.485992 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.466402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41723 (local address 127.31.26.194:44181)
0927 18:46:07.467226 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.467279 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.468898 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.469622 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.469630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.470137 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:07.470697 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.470715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.471827 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.471831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.473060 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.473071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.474209 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.474214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.474428 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.475120 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:07.475146 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:07.477602 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:07.481839 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:07.481846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:07.481861 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:07.482331 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:07.482772 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:07.482775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:07.482778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:07.482956 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:07.483618 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:07.483624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:07.483964 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 18:46:07.484218 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 18:46:07.484674 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":327,"threads_started":1}
I20260927 18:46:07.487619 10067 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41723
I20260927 18:46:07.487993 10067 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:07.488955 10067 heartbeater.cc:511] Master 127.31.26.254:41723 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:46:07.490643 9744 ts_manager.cc:194] Registered new tserver with Master: 1568e8ea1f7544409438c844722f42b1 (127.31.26.194:39429)
I20260927 18:46:07.491469 9744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:44181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:46:07.520577 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41723
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:07.857988 10073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:07.858269 10073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:07.858415 10073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:07.865983 10073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:07.866106 10073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:07.866273 10073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:46:07.876463 10073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:07.877921 10073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:07.879333 10073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:07.893069 10081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.893075 10078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:07.893121 10079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:07.894269 10073 server_base.cc:1034] running on GCE node
I20260927 18:46:07.894997 10073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:07.896059 10073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:07.897294 10073 hybrid_clock.cc:648] HybridClock initialized: now 1790534767897235 us; error 54 us; skew 500 ppm
I20260927 18:46:07.897653 10073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:07 dist-test-slave-rfd5 krb5kdc[9699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:07.900784 10073 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:46:07.902351 10073 webserver.cc:460] Webserver started at http://127.31.26.195:35689/ using document root <none> and password file <none>
I20260927 18:46:07.903107 10073 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:07.903255 10073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:07.903577 10073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:07.906618 10073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "45e7d5548b9c4ead997a59db9127ed6e"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-rfd5"
server_key: "669d92c6287345d4eea74c833065c43e"
server_key_iv: "e340edb805b1960532f2256d351a3d05"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.907390 10073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "45e7d5548b9c4ead997a59db9127ed6e"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-rfd5"
server_key: "669d92c6287345d4eea74c833065c43e"
server_key_iv: "e340edb805b1960532f2256d351a3d05"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.912738 10073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:46:07.917392 10088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.919085 10073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 18:46:07.919286 10073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "45e7d5548b9c4ead997a59db9127ed6e"
format_stamp: "Formatted at 2026-09-27 18:46:07 on dist-test-slave-rfd5"
server_key: "669d92c6287345d4eea74c833065c43e"
server_key_iv: "e340edb805b1960532f2256d351a3d05"
server_key_version: "encryptionkey@0"
I20260927 18:46:07.919504 10073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:07.943104 10073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:07.947502 10073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:07.947618 9933 heartbeater.cc:503] Master 127.31.26.254:41723 was elected leader, sending a full tablet report...
I20260927 18:46:07.947857 10073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:07.949124 10073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:07.950594 10073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:07.950671 10073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.950755 10073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:07.950799 10073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:07.984243 10073 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:34529
I20260927 18:46:07.984333 10200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:34529 every 8 connection(s)
I20260927 18:46:07.985687 10073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:07.994314 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 10073
I20260927 18:46:07.994539 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:07.995066 31851 external_mini_cluster.cc:1428] Setting key 4cb7b8ec02596ffec48d66a91a4fee14
Sep 27 18:46:07 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:08.006094 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.988201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:36669 (local address 127.31.26.254:41723)
0927 18:46:07.988654 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:07.988662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:07.989679 (+ 1017us) server_negotiation.cc:407] Connection header received
0927 18:46:07.991419 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:07.991423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:07.991492 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:07.991596 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:07.993063 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.993919 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.996074 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.996591 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.996722 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.000219 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.000225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.000228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:08.000262 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.002854 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.003624 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.003631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.003635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.003764 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.004213 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.004220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.004223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.004499 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.004674 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.005471 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.005709 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":195,"threads_started":1}
I20260927 18:46:08.007124 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:07.988769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41723 (local address 127.31.26.195:36669)
0927 18:46:07.989523 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:07.989567 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:07.991171 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:07.991803 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:07.991811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:07.992334 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:07.992897 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.992918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.994092 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.994098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.995922 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:07.995934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:07.996810 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:07.996820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:07.997119 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:07.997841 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:07.997865 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:07.999984 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:08.003004 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:08.003014 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:08.003041 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:08.003499 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:08.003887 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:08.003892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:08.003896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:08.004101 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:08.004643 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:08.004652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:08.005020 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.005381 (+ 361us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.005952 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":320,"threads_started":1}
I20260927 18:46:08.008533 10201 heartbeater.cc:347] Connected to a master server at 127.31.26.254:41723
I20260927 18:46:08.008874 10201 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:08.009689 10201 heartbeater.cc:511] Master 127.31.26.254:41723 requested a full tablet report, sending...
I20260927 18:46:08.011152 9744 ts_manager.cc:194] Registered new tserver with Master: 45e7d5548b9c4ead997a59db9127ed6e (127.31.26.195:34529)
I20260927 18:46:08.011970 9744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:36669
I20260927 18:46:08.023030 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:08.040385 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.027617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.31.26.254:41723)
0927 18:46:08.027858 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.027866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.028134 (+ 268us) server_negotiation.cc:407] Connection header received
0927 18:46:08.028280 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.028284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.028379 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.028489 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.030023 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.030760 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.032017 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.032479 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.032584 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.035083 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.035088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.035091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:08.035125 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.037886 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.038517 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.038522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.038525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.038646 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.038975 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.038979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.038982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.039377 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.039551 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.039858 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.040019 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:46:08.044682 9744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:08.051435 9744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:08.074179 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.063807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34037 (local address 127.0.0.1:53306)
0927 18:46:08.064955 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.065029 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.065234 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.066425 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.066431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.066460 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.066967 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.066978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.069049 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.069055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.071281 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.071293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.073115 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.073123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.073316 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.073369 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.073483 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.073694 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":13,"thread_start_us":736,"threads_started":1}
I20260927 18:46:08.074638 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.063762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.31.26.194:39429)
0927 18:46:08.064494 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.064506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.065156 (+ 650us) server_negotiation.cc:407] Connection header received
0927 18:46:08.065484 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.065495 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.065760 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.065938 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.067140 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.068788 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.071753 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.073231 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.073362 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.073595 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.074141 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.074321 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":427,"threads_started":1}
I20260927 18:46:08.074983 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.064337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.31.26.193:34037)
0927 18:46:08.065849 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.065861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.065888 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:46:08.065976 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.065984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.066222 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.067140 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.067373 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.068865 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.071434 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.072884 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.073032 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.074407 (+ 1375us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.074492 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.074657 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1343,"thread_start_us":1192,"threads_started":1}
I20260927 18:46:08.075001 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.066151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:34529 (local address 127.0.0.1:43472)
0927 18:46:08.066950 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.066969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.067155 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.067962 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.067965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.067983 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.068253 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.068260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.070125 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.070130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.072124 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.072134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.073870 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.073877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.074044 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.074081 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.074187 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.074329 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":1,"thread_start_us":497,"threads_started":1}
I20260927 18:46:08.075241 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.065737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.31.26.195:34529)
0927 18:46:08.066408 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.066417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.067061 (+ 644us) server_negotiation.cc:407] Connection header received
0927 18:46:08.067334 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.067342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.067569 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.067725 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.068425 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.069947 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.072266 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.073339 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.073480 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.073720 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.074689 (+ 969us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.074874 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":513,"threads_started":1}
I20260927 18:46:08.075711 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.063652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:39429 (local address 127.0.0.1:57256)
0927 18:46:08.064955 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.065029 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.065234 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.066261 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.066269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.066298 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.066897 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.066915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.068941 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.068947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.071594 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.071607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.073736 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.073743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.073918 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.073947 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.075264 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.075440 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":64,"spinlock_wait_cycles":4864,"thread_start_us":997,"threads_started":1}
I20260927 18:46:08.079166 10002 tablet_service.cc:1467] Processing CreateTablet for tablet e6397f7b129a435ea6d5e81b92901c2d (DEFAULT_TABLE table=test-table [id=b3f014b7bc7248a9a2d79290d00a571a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:08.079169 9868 tablet_service.cc:1467] Processing CreateTablet for tablet e6397f7b129a435ea6d5e81b92901c2d (DEFAULT_TABLE table=test-table [id=b3f014b7bc7248a9a2d79290d00a571a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:08.079169 10136 tablet_service.cc:1467] Processing CreateTablet for tablet e6397f7b129a435ea6d5e81b92901c2d (DEFAULT_TABLE table=test-table [id=b3f014b7bc7248a9a2d79290d00a571a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:08.080360 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6397f7b129a435ea6d5e81b92901c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:08.080520 10136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6397f7b129a435ea6d5e81b92901c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:08.080502 9868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6397f7b129a435ea6d5e81b92901c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:08.093885 10220 tablet_bootstrap.cc:492] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Bootstrap starting.
I20260927 18:46:08.094903 10222 tablet_bootstrap.cc:492] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Bootstrap starting.
I20260927 18:46:08.098783 10222 tablet_bootstrap.cc:654] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:08.099071 10220 tablet_bootstrap.cc:654] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:08.099741 10221 tablet_bootstrap.cc:492] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Bootstrap starting.
I20260927 18:46:08.100178 10222 log.cc:824] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:08.100274 10220 log.cc:824] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:08.103272 10222 tablet_bootstrap.cc:492] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: No bootstrap required, opened a new log
I20260927 18:46:08.103579 10222 ts_tablet_manager.cc:1397] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:46:08.103634 10221 tablet_bootstrap.cc:654] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:08.103705 10220 tablet_bootstrap.cc:492] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: No bootstrap required, opened a new log
I20260927 18:46:08.103953 10220 ts_tablet_manager.cc:1397] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260927 18:46:08.105298 10221 log.cc:824] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:08.108935 10221 tablet_bootstrap.cc:492] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: No bootstrap required, opened a new log
I20260927 18:46:08.109256 10221 ts_tablet_manager.cc:1397] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:46:08.112602 10222 raft_consensus.cc:348] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.113176 10222 raft_consensus.cc:374] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:08.113312 10222 raft_consensus.cc:729] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd0802728e04f58965d1ef0774ddb83, State: Initialized, Role: FOLLOWER
I20260927 18:46:08.114084 10220 raft_consensus.cc:348] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.114183 10222 consensus_queue.cc:260] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.114537 10220 raft_consensus.cc:374] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:08.114658 10220 raft_consensus.cc:729] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1568e8ea1f7544409438c844722f42b1, State: Initialized, Role: FOLLOWER
I20260927 18:46:08.115386 10220 consensus_queue.cc:260] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.118855 10067 heartbeater.cc:503] Master 127.31.26.254:41723 was elected leader, sending a full tablet report...
I20260927 18:46:08.119700 10220 ts_tablet_manager.cc:1428] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260927 18:46:08.120350 10222 ts_tablet_manager.cc:1428] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260927 18:46:08.123996 10221 raft_consensus.cc:348] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.124493 10221 raft_consensus.cc:374] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:08.124648 10221 raft_consensus.cc:729] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e7d5548b9c4ead997a59db9127ed6e, State: Initialized, Role: FOLLOWER
I20260927 18:46:08.125383 10221 consensus_queue.cc:260] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.126494 10201 heartbeater.cc:503] Master 127.31.26.254:41723 was elected leader, sending a full tablet report...
I20260927 18:46:08.127139 10221 ts_tablet_manager.cc:1428] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
W20260927 18:46:08.167577 9934 tablet.cc:2369] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:08.215137 10068 tablet.cc:2369] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:08.237325 10202 tablet.cc:2369] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:08.376200 10227 raft_consensus.cc:482] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:08.376508 10227 raft_consensus.cc:504] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.378571 10227 leader_election.cc:288] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd0802728e04f58965d1ef0774ddb83 (127.31.26.193:34037), 45e7d5548b9c4ead997a59db9127ed6e (127.31.26.195:34529)
I20260927 18:46:08.388197 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.379276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:57383 (local address 127.31.26.193:34037)
0927 18:46:08.379517 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.379524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.379715 (+ 191us) server_negotiation.cc:407] Connection header received
0927 18:46:08.380065 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.380071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.380159 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.380227 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.381075 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.382236 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.385586 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.386843 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.386995 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.387071 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.387803 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.387941 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:46:08.388195 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.379147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:48395 (local address 127.31.26.195:34529)
0927 18:46:08.379396 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.379402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.380228 (+ 826us) server_negotiation.cc:407] Connection header received
0927 18:46:08.380387 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.380395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.380484 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.380612 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:08.381285 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.382340 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.385586 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.386815 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.386996 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.387071 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.387803 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.387953 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:46:08.388490 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.378955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:34529 (local address 127.31.26.194:48395)
0927 18:46:08.380114 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.380136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.380249 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.380634 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.380638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.380656 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.381074 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.381109 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.382517 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.382520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.385423 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.385439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.387122 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.387127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.387471 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.387658 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.387759 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.387985 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":408,"thread_start_us":462,"threads_started":1}
I20260927 18:46:08.388515 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.378751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34037 (local address 127.31.26.194:57383)
0927 18:46:08.379580 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:08.379604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:08.379853 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:08.380440 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:08.380443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:08.380462 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:08.380868 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.380891 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.382515 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.382520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.385356 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:08.385374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.387115 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.387122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:08.387471 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.387659 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 18:46:08.387758 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:46:08.387985 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":377,"thread_start_us":981,"threads_started":1}
I20260927 18:46:08.389616 9888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6397f7b129a435ea6d5e81b92901c2d" candidate_uuid: "1568e8ea1f7544409438c844722f42b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd0802728e04f58965d1ef0774ddb83" is_pre_election: true
I20260927 18:46:08.389740 10156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6397f7b129a435ea6d5e81b92901c2d" candidate_uuid: "1568e8ea1f7544409438c844722f42b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e7d5548b9c4ead997a59db9127ed6e" is_pre_election: true
I20260927 18:46:08.390166 9888 raft_consensus.cc:2454] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1568e8ea1f7544409438c844722f42b1 in term 0.
I20260927 18:46:08.390272 10156 raft_consensus.cc:2454] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1568e8ea1f7544409438c844722f42b1 in term 0.
I20260927 18:46:08.390875 9955 leader_election.cc:302] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1568e8ea1f7544409438c844722f42b1, edd0802728e04f58965d1ef0774ddb83; no voters:
I20260927 18:46:08.391297 10227 raft_consensus.cc:2781] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:08.391444 10227 raft_consensus.cc:482] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:08.391556 10227 raft_consensus.cc:3037] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:08.393831 10227 raft_consensus.cc:504] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.394469 10227 leader_election.cc:288] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [CANDIDATE]: Term 1 election: Requested vote from peers edd0802728e04f58965d1ef0774ddb83 (127.31.26.193:34037), 45e7d5548b9c4ead997a59db9127ed6e (127.31.26.195:34529)
I20260927 18:46:08.394956 9888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6397f7b129a435ea6d5e81b92901c2d" candidate_uuid: "1568e8ea1f7544409438c844722f42b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd0802728e04f58965d1ef0774ddb83"
I20260927 18:46:08.395054 10156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6397f7b129a435ea6d5e81b92901c2d" candidate_uuid: "1568e8ea1f7544409438c844722f42b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e7d5548b9c4ead997a59db9127ed6e"
I20260927 18:46:08.395191 9888 raft_consensus.cc:3037] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:08.395241 10156 raft_consensus.cc:3037] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:08.398424 10156 raft_consensus.cc:2454] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1568e8ea1f7544409438c844722f42b1 in term 1.
I20260927 18:46:08.398422 9888 raft_consensus.cc:2454] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1568e8ea1f7544409438c844722f42b1 in term 1.
I20260927 18:46:08.398893 9955 leader_election.cc:302] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1568e8ea1f7544409438c844722f42b1, 45e7d5548b9c4ead997a59db9127ed6e, edd0802728e04f58965d1ef0774ddb83; no voters:
I20260927 18:46:08.399271 10227 raft_consensus.cc:2781] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:08.399679 10227 raft_consensus.cc:686] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1568e8ea1f7544409438c844722f42b1, State: Running, Role: LEADER
I20260927 18:46:08.400153 10227 consensus_queue.cc:237] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } }
I20260927 18:46:08.405278 9744 catalog_manager.cc:5495] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1568e8ea1f7544409438c844722f42b1 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "1568e8ea1f7544409438c844722f42b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1568e8ea1f7544409438c844722f42b1" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:08.444841 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.439592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.31.26.194:39429)
0927 18:46:08.439814 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.439823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.439846 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:46:08.440101 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.440105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.440191 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.440317 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:08.440645 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.441442 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.442766 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.443163 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.443259 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.443781 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:08.444336 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:08.444452 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.444556 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.444625 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:46:08.453379 10156 raft_consensus.cc:1261] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 1 FOLLOWER]: Refusing update from remote peer 1568e8ea1f7544409438c844722f42b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:08.453379 9888 raft_consensus.cc:1261] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 1 FOLLOWER]: Refusing update from remote peer 1568e8ea1f7544409438c844722f42b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:08.454438 10231 consensus_queue.cc:1035] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd0802728e04f58965d1ef0774ddb83" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:08.454825 10227 consensus_queue.cc:1035] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e7d5548b9c4ead997a59db9127ed6e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:08.471316 10234 mvcc.cc:205] Tried to move back new op lower bound from 7334030411574554624 to 7334030411371040768. Current Snapshot: MvccSnapshot[applied={T|T < 7334030411574554624}]
I20260927 18:46:08.490880 9743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:08.491287 9743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:08.495571 9743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c2ef4d530c5474dacc8f8836dae3aed: Sending DeleteTablet for 3 replicas of tablet e6397f7b129a435ea6d5e81b92901c2d
I20260927 18:46:08.496869 9868 tablet_service.cc:1514] Processing DeleteTablet for tablet e6397f7b129a435ea6d5e81b92901c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:08 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:53306
I20260927 18:46:08.496969 10002 tablet_service.cc:1514] Processing DeleteTablet for tablet e6397f7b129a435ea6d5e81b92901c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:08 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:57256
I20260927 18:46:08.496948 10136 tablet_service.cc:1514] Processing DeleteTablet for tablet e6397f7b129a435ea6d5e81b92901c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:08 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:43472
I20260927 18:46:08.499297 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9805
I20260927 18:46:08.507968 10244 tablet_replica.cc:330] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: stopping tablet replica
I20260927 18:46:08.508637 10244 raft_consensus.cc:2229] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:08.511310 10244 raft_consensus.cc:2258] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:08.515275 10244 ts_tablet_manager.cc:1905] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:08.519495 10246 tablet_replica.cc:330] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: stopping tablet replica
I20260927 18:46:08.519647 10245 tablet_replica.cc:330] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: stopping tablet replica
I20260927 18:46:08.520010 10246 raft_consensus.cc:2229] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:08.520004 10245 raft_consensus.cc:2229] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:08.520516 10245 raft_consensus.cc:2258] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:08.520663 10246 raft_consensus.cc:2258] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:08.523062 10246 ts_tablet_manager.cc:1905] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:08.524118 10245 ts_tablet_manager.cc:1905] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:46:08 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:46:08.530714 10246 ts_tablet_manager.cc:1918] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:08.530918 10246 log.cc:1196] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/e6397f7b129a435ea6d5e81b92901c2d
I20260927 18:46:08.531504 10246 ts_tablet_manager.cc:1939] T e6397f7b129a435ea6d5e81b92901c2d P 1568e8ea1f7544409438c844722f42b1: Deleting consensus metadata
I20260927 18:46:08.533406 9731 catalog_manager.cc:4841] TS 1568e8ea1f7544409438c844722f42b1 (127.31.26.194:39429): tablet e6397f7b129a435ea6d5e81b92901c2d (table test-table [id=b3f014b7bc7248a9a2d79290d00a571a]) successfully deleted
I20260927 18:46:08.534340 10245 ts_tablet_manager.cc:1918] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:08.534509 10245 log.cc:1196] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/e6397f7b129a435ea6d5e81b92901c2d
I20260927 18:46:08.535020 10245 ts_tablet_manager.cc:1939] T e6397f7b129a435ea6d5e81b92901c2d P 45e7d5548b9c4ead997a59db9127ed6e: Deleting consensus metadata
I20260927 18:46:08.536075 10244 ts_tablet_manager.cc:1918] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:08.536263 10244 log.cc:1196] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/e6397f7b129a435ea6d5e81b92901c2d
I20260927 18:46:08.536520 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.500755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.31.26.193:34037)
0927 18:46:08.500938 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.500945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.513924 (+ 12979us) server_negotiation.cc:407] Connection header received
0927 18:46:08.514469 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.514476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.514564 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.514650 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.516608 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.517816 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.519914 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.520563 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.520667 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.526812 (+ 6145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.526831 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.526851 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:08.526898 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.531547 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.532614 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.532624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.532628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.532799 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.533904 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.533909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.533912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.534427 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.534889 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.535573 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.535841 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:46:08.536896 9731 catalog_manager.cc:4841] TS 45e7d5548b9c4ead997a59db9127ed6e (127.31.26.195:34529): tablet e6397f7b129a435ea6d5e81b92901c2d (table test-table [id=b3f014b7bc7248a9a2d79290d00a571a]) successfully deleted
I20260927 18:46:08.536962 10244 ts_tablet_manager.cc:1939] T e6397f7b129a435ea6d5e81b92901c2d P edd0802728e04f58965d1ef0774ddb83: Deleting consensus metadata
I20260927 18:46:08.538690 9928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:08.539515 9728 catalog_manager.cc:4841] TS edd0802728e04f58965d1ef0774ddb83 (127.31.26.193:34037): tablet e6397f7b129a435ea6d5e81b92901c2d (table test-table [id=b3f014b7bc7248a9a2d79290d00a571a]) successfully deleted
I20260927 18:46:08.628806 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9805
I20260927 18:46:08.647203 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9939
Sep 27 18:46:08 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:46:08.664852 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.648031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.31.26.194:39429)
0927 18:46:08.648219 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.648227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.648252 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:46:08.648472 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.648477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.648583 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.648692 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.650014 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.650837 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.652100 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.652617 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.652760 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.657676 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.657698 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.657720 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:08.657786 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.661781 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.662538 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.662548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.662552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.662743 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.663133 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.663138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.663141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.663607 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.663930 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.664228 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.664455 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:46:08.665620 10062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:08.779165 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9939
I20260927 18:46:08.797998 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 10073
Sep 27 18:46:08 dist-test-slave-rfd5 krb5kdc[9699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:46:08.813796 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:08.798771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.31.26.195:34529)
0927 18:46:08.798952 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:08.798960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:08.799023 (+ 63us) server_negotiation.cc:407] Connection header received
0927 18:46:08.799280 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:08.799290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:08.799366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:08.799470 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:08.800727 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.801523 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.802790 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:08.803187 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:08.803317 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:08.807903 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:08.807920 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:08.807937 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:08.807976 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:08.810863 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.811610 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.811617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.811620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.811773 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:08.812175 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:08.812181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:08.812186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:08.812586 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:08.812882 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:08.813230 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 18:46:08.813439 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:46:08.814472 10196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:08.904461 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 10073
I20260927 18:46:08.921212 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 9712
I20260927 18:46:08.922389 9774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:09.031267 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 9712
2026-09-27T18:46:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4537 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:09 dist-test-slave-rfd5 krb5kdc[10254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:09 dist-test-slave-rfd5 krb5kdc[10254](info): set up 2 sockets
Sep 27 18:46:09 dist-test-slave-rfd5 krb5kdc[10254](info): commencing operation
krb5kdc: starting...
W20260927 18:46:10.142040 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:46:10 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/rangeradmin_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/rangeradmin_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/rangerlookup_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/rangerlookup_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
I20260927 18:46:10.474145 31851 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskk9LG2U/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/postgres -l logfile start
2026-09-27 18:46:13.115 UTC [10284] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:46:13.115 UTC [10284] LOG: listening on IPv4 address "127.31.26.212", port 46023
2026-09-27 18:46:13.119 UTC [10284] LOG: listening on Unix socket "/tmp/.s.PGSQL.46023"
2026-09-27 18:46:13.141 UTC [10287] LOG: database system was shut down at 2026-09-27 18:46:11 UTC
2026-09-27 18:46:13.145 UTC [10284] LOG: database system is ready to accept connections
I20260927 18:46:14.127674 31851 mini_postgres.cc:96] Postgres bound to 46023
2026-09-27 18:46:14.151 UTC [10295] FATAL: database "slave" does not exist
127.31.26.212:46023 - accepting connections
I20260927 18:46:14.151922 31851 mini_ranger.cc:161] Starting Ranger...
I20260927 18:46:14.179955 31851 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:46:14.710997 31851 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:46:14.711143 31851 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-admin
2026-09-27 18:46:15,012 [I] DB FLAVOR :POSTGRES
2026-09-27 18:46:15,013 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:46:15,013 [I] Checking connection..
2026-09-27 18:46:15,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:46:15,260 [I] Checking connection passed.
2026-09-27 18:46:15,260 [I] --------- Verifying version history table ---------
2026-09-27 18:46:15,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:46:15,513 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:46:15,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:46:15,782 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:46:15,782 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:46:15,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:46:16.032 UTC [10359] WARNING: there is no transaction in progress
2026-09-27 18:46:16,039 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:46:16,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:46:16,301 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:46:16,301 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:46:16,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:46:16,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:46:16,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rfd5.c.gcp-upstream.internal','N') ;"
2026-09-27 18:46:17,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:46:17,314 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:46:17,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:46:17,569 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:46:17,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:46:17,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:46:18,068 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:46:18,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:46:19.067 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.083 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.100 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.114 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.130 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.148 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.215 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.222 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.237 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.248 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.259 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.274 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.284 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.290 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.297 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19.304 UTC [10467] WARNING: there is no transaction in progress
2026-09-27 18:46:19,982 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:46:19,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:46:20,229 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:46:20,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:46:20,494 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:46:20,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:46:20,739 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:46:20,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:46:21,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:46:21,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rfd5.c.gcp-upstream.internal';"
2026-09-27 18:46:21,491 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:46:21,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:46:21,724 [I] DB_PATCHES have already been applied
I20260927 18:46:21.730669 31851 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-admin:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskk9LG2U/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:46:21.730762 31851 mini_ranger.cc:193] Using host: 127.31.26.212
I20260927 18:46:21.745630 31851 mini_ranger.cc:227] Ranger admin URL: http://127.31.26.212:37323
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:46:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:46:22.883342 31851 mini_ranger.cc:160] Time spent starting Ranger: real 8.731s user 0.000s sys 0.041s
Sep 27, 2026 6:46:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:46:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:46:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:46:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:46:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:46:46.111102 31851 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/rangerkms_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/rangerkms_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.26.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.26.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:46:46 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534806, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:46:46.301596 31851 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:46:46.323341 31851 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:46:47.256778 31851 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:46:47.256999 31851 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms
I20260927 18:46:47.257174 31851 mini_ranger_kms.cc:227] Using postgres at 127.31.26.212:46023
2026-09-27 18:46:47,688 [I] DB FLAVOR :POSTGRES
2026-09-27 18:46:47,688 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:46:47,688 [I] Checking connection
2026-09-27 18:46:47,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:46:47,943 [I] connection success
2026-09-27 18:46:47,943 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:46:47,943 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:46:47,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:46:48,191 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:46:48,191 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:46:48,191 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:46:48,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:46023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:46:48,461 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:46:49.029891 31851 mini_ranger_kms.cc:310] Created kms service
I20260927 18:46:49.252084 31851 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:46:49.305299 31851 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:46:49 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534806, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:46:49.733227 31851 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:46:49.733575 31851 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms:/tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:46:49.733633 31851 mini_ranger_kms.cc:240] Using host: 127.31.26.212
I20260927 18:46:49.754593 31851 mini_ranger_kms.cc:276] Ranger KMS PID: 10648
I20260927 18:46:49.754799 31851 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.26.212:48761
Sep 27, 2026 6:46:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:46:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:46:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:46:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:46:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:46:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 18:46:51 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534811, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.26.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:52.089912 31851 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.788s user 0.127s sys 0.327s
I20260927 18:46:52.090183 31851 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:46:52.090396 31851 mini_ranger_kms.cc:401] 127.31.26.212:48761/kms/v1/keys
Sep 27 18:46:52 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534811, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.26.212@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
Sep 27 18:46:52 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534811, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.26.212@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
Sep 27 18:46:55 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:46:55.201602 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:39243
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:39243
--ranger_config_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:48761/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:55.555886 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:55.556198 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:55.556360 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:55.563766 10698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:55.563892 10698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:55.563984 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:55.564095 10698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:55.564181 10698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:55.574535 10698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.26.212:48761/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:39243
--ranger_config_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:39243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:55.575939 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:55.577384 10698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:55.591035 10703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:55.591086 10706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:55.590991 10704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:55.596853 10698 server_base.cc:1034] running on GCE node
I20260927 18:46:55.597847 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:55.599472 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:55.600708 10698 hybrid_clock.cc:648] HybridClock initialized: now 1790534815600638 us; error 57 us; skew 500 ppm
I20260927 18:46:55.601192 10698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:55 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:55.609468 10698 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:46:55.611272 10698 webserver.cc:460] Webserver started at http://127.31.26.254:35787/ using document root <none> and password file <none>
I20260927 18:46:55.611985 10698 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:55.612084 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:55.612350 10698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:46:55 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:46:55.672807 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "96957facd6fb4b7a992574bf56ca8e65"
format_stamp: "Formatted at 2026-09-27 18:46:55 on dist-test-slave-rfd5"
server_key: "49db0db2a242655c089a50dfd45c2b3b"
server_key_iv: "72713a5f5e2d8cdb70e1ec21afad2d0e"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:55.673756 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "96957facd6fb4b7a992574bf56ca8e65"
format_stamp: "Formatted at 2026-09-27 18:46:55 on dist-test-slave-rfd5"
server_key: "49db0db2a242655c089a50dfd45c2b3b"
server_key_iv: "72713a5f5e2d8cdb70e1ec21afad2d0e"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:55.679762 10698 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:46:55.700829 10713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:55.702518 10698 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:46:55.702715 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "96957facd6fb4b7a992574bf56ca8e65"
format_stamp: "Formatted at 2026-09-27 18:46:55 on dist-test-slave-rfd5"
server_key: "49db0db2a242655c089a50dfd45c2b3b"
server_key_iv: "72713a5f5e2d8cdb70e1ec21afad2d0e"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:55.702927 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:55.723953 10698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:55.728601 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:55.729024 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:55.750336 10698 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:39243
I20260927 18:46:55.750355 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:39243 every 8 connection(s)
I20260927 18:46:55.751907 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:55.756137 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:55.761576 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 10698
I20260927 18:46:55.761792 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:46:55.762599 31851 external_mini_cluster.cc:1428] Setting key 63f1279888684f7622b07af5fe760111
I20260927 18:46:55.768635 10765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65: Bootstrap starting.
I20260927 18:46:55.772575 10765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:55.774219 10765 log.cc:824] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:55.777674 10765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65: No bootstrap required, opened a new log
Sep 27 18:46:55 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:55.787513 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:55.764752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.31.26.254:39243)
0927 18:46:55.765660 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:55.765673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:55.765718 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:46:55.767516 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:55.767540 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:55.767981 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:55.768331 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:55.769995 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:55.771397 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:55.773661 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:55.774382 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:55.774602 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:55.779201 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:55.779216 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:55.779237 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:55.779291 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:55.783224 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:55.783991 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:55.783998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:55.784006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:55.784176 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:55.784607 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:55.784611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:55.784613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:55.785248 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:55.785597 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:55.786070 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 18:46:55.786496 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":470,"threads_started":1}
I20260927 18:46:55.787607 10765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96957facd6fb4b7a992574bf56ca8e65" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39243 } }
I20260927 18:46:55.788091 10765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:55.788225 10765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96957facd6fb4b7a992574bf56ca8e65, State: Initialized, Role: FOLLOWER
I20260927 18:46:55.788846 10765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96957facd6fb4b7a992574bf56ca8e65" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39243 } }
I20260927 18:46:55.789601 10765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:55.789743 10765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:55.789933 10765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:55.792315 10765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96957facd6fb4b7a992574bf56ca8e65" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39243 } }
I20260927 18:46:55.792797 10765 leader_election.cc:302] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96957facd6fb4b7a992574bf56ca8e65; no voters:
I20260927 18:46:55.793398 10765 leader_election.cc:288] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:55.793538 10770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:55.794330 10770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [term 1 LEADER]: Becoming Leader. State: Replica: 96957facd6fb4b7a992574bf56ca8e65, State: Running, Role: LEADER
I20260927 18:46:55.794957 10770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96957facd6fb4b7a992574bf56ca8e65" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39243 } }
I20260927 18:46:55.795634 10765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:55.797752 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96957facd6fb4b7a992574bf56ca8e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96957facd6fb4b7a992574bf56ca8e65" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39243 } } }
I20260927 18:46:55.798189 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:55.803630 10772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96957facd6fb4b7a992574bf56ca8e65. Latest consensus state: current_term: 1 leader_uuid: "96957facd6fb4b7a992574bf56ca8e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96957facd6fb4b7a992574bf56ca8e65" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 39243 } } }
I20260927 18:46:55.806945 10772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:55.807878 10779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:55.812863 10779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:55.813743 10765 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:46:55.831043 10779 catalog_manager.cc:1347] Generated new cluster ID: df488f6c838b4761a14cf4d84453e296
I20260927 18:46:55.831231 10779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:55.862311 10779 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:55.864490 10779 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:55.877454 10779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65: Generated new TSK 0
I20260927 18:46:55.878641 10779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:46:56.659104 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:48761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39243
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:46:56 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:46:57.099797 10810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:57.100160 10810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:57.100332 10810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:57.110252 10810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:57.110412 10810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:57.110611 10810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:46:57.124763 10810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.26.212:48761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:57.126573 10810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:57.128463 10810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:57.150108 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:57.150426 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:57.150141 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:57.151518 10810 server_base.cc:1034] running on GCE node
I20260927 18:46:57.152282 10810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:57.155556 10810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:57.156893 10810 hybrid_clock.cc:648] HybridClock initialized: now 1790534817156783 us; error 103 us; skew 500 ppm
I20260927 18:46:57.157356 10810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:57 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:57.161510 10810 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:46:57.166891 10810 webserver.cc:460] Webserver started at http://127.31.26.193:42037/ using document root <none> and password file <none>
I20260927 18:46:57.167986 10810 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:57.168179 10810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:57.168617 10810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:46:57 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:46:57.195499 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "4cf3594b63574342a2c7504d7c57d46b"
format_stamp: "Formatted at 2026-09-27 18:46:57 on dist-test-slave-rfd5"
server_key: "98207b0751f4a750868fa06cc758800d"
server_key_iv: "67949aacd5895838d1918a00cdf754b8"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:57.196694 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "4cf3594b63574342a2c7504d7c57d46b"
format_stamp: "Formatted at 2026-09-27 18:46:57 on dist-test-slave-rfd5"
server_key: "98207b0751f4a750868fa06cc758800d"
server_key_iv: "67949aacd5895838d1918a00cdf754b8"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:57.204622 10810 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 18:46:57.237816 10830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:57.242895 10810 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.001s sys 0.012s
I20260927 18:46:57.243227 10810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "4cf3594b63574342a2c7504d7c57d46b"
format_stamp: "Formatted at 2026-09-27 18:46:57 on dist-test-slave-rfd5"
server_key: "98207b0751f4a750868fa06cc758800d"
server_key_iv: "67949aacd5895838d1918a00cdf754b8"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:57.243623 10810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:57.282382 10810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:57.288184 10810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:57.289006 10810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:57.290306 10810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:57.292526 10810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:57.292627 10810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:57.292809 10810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:57.292893 10810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:57.369910 10810 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:35135
I20260927 18:46:57.369910 10942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:35135 every 8 connection(s)
I20260927 18:46:57.371379 10810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:57.375741 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 10810
I20260927 18:46:57.375958 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:57.376736 31851 external_mini_cluster.cc:1428] Setting key b20a512d7bde8d7aaca58a46ed72aa27
Sep 27 18:46:57 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Sep 27 18:46:57 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534817, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:46:57.439688 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:57.390145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:39243 (local address 127.31.26.193:51829)
0927 18:46:57.397788 (+ 7643us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:57.397857 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:57.405303 (+ 7446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:57.408979 (+ 3676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:57.408996 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:57.409829 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:57.410866 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:57.410902 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:57.412624 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:57.412634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:57.423126 (+ 10492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:57.423161 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:57.424617 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:57.424627 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:57.424962 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:57.425836 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:57.425869 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:57.428552 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:57.432871 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:57.432882 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:57.432904 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:57.434755 (+ 1851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:57.435276 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:57.435284 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:57.435290 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:57.435647 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:57.436445 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:57.436456 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:57.436994 (+ 538us) client_negotiation.cc:769] Sending connection context
0927 18:46:57.437362 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 18:46:57.437978 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7297,"thread_start_us":7009,"threads_started":1}
I20260927 18:46:57.441339 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:57.390785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:51829 (local address 127.31.26.254:39243)
0927 18:46:57.408195 (+ 17410us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:57.408202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:57.408226 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:46:57.408293 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:57.408299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:57.408382 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:57.408511 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:57.411133 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:57.412400 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:57.423375 (+ 10975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:57.424277 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:57.424464 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:57.428849 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:57.428855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:57.428858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:57.428904 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:57.432658 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:57.434935 (+ 2277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:57.434944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:57.434951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:57.435127 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:57.435799 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:57.435807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:57.435814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:57.436214 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:57.436367 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:57.440509 (+ 4142us) server_negotiation.cc:299] Negotiation successful
0927 18:46:57.440824 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17273,"thread_start_us":17067,"threads_started":1}
I20260927 18:46:57.443014 10943 heartbeater.cc:347] Connected to a master server at 127.31.26.254:39243
I20260927 18:46:57.443364 10943 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:57.444037 10943 heartbeater.cc:511] Master 127.31.26.254:39243 requested a full tablet report, sending...
I20260927 18:46:57.446557 10730 ts_manager.cc:194] Registered new tserver with Master: 4cf3594b63574342a2c7504d7c57d46b (127.31.26.193:35135)
I20260927 18:46:57.448800 10730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:51829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:46:57.476830 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:48761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39243
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:46:57 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
W20260927 18:46:57.992640 10949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:57.993184 10949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:57.993407 10949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:58.006108 10949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:58.006336 10949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:58.006567 10949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:46:58.024658 10949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.26.212:48761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:58.026582 10949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:58.028838 10949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:58.062173 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:58.062513 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:58.064816 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:58.065176 10949 server_base.cc:1034] running on GCE node
I20260927 18:46:58.065927 10949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:58.066957 10949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:58.068197 10949 hybrid_clock.cc:648] HybridClock initialized: now 1790534818068136 us; error 59 us; skew 500 ppm
I20260927 18:46:58.068576 10949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:58 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:58.072383 10949 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:46:58.074190 10949 webserver.cc:460] Webserver started at http://127.31.26.194:35423/ using document root <none> and password file <none>
I20260927 18:46:58.074975 10949 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:58.075112 10949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:58.075445 10949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:46:58 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:46:58.093823 10949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "3beccb6d607c4886b45cee96f01f91fb"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-rfd5"
server_key: "4ed3dfb410f4a17f4d6d2aed0b86891a"
server_key_iv: "33281357d8a33195d4065ae9bc100063"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:58.095021 10949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "3beccb6d607c4886b45cee96f01f91fb"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-rfd5"
server_key: "4ed3dfb410f4a17f4d6d2aed0b86891a"
server_key_iv: "33281357d8a33195d4065ae9bc100063"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:58.103313 10949 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260927 18:46:58.137838 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:58.143327 10949 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.001s sys 0.015s
I20260927 18:46:58.143582 10949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "3beccb6d607c4886b45cee96f01f91fb"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-rfd5"
server_key: "4ed3dfb410f4a17f4d6d2aed0b86891a"
server_key_iv: "33281357d8a33195d4065ae9bc100063"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:58.143870 10949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:58.171430 10949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:58.177424 10949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:58.177796 10949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:58.179062 10949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:58.180900 10949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:58.180998 10949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:58.181149 10949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:58.181214 10949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 18:46:58.228143 10949 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:37163
I20260927 18:46:58.228164 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:37163 every 8 connection(s)
I20260927 18:46:58.229940 10949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:58.234338 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 10949
I20260927 18:46:58.234511 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:58.235044 31851 external_mini_cluster.cc:1428] Setting key 64f9f59e3ade8b55674700c721aca330
Sep 27 18:46:58 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:58.262688 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.236154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:36931 (local address 127.31.26.254:39243)
0927 18:46:58.243894 (+ 7740us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.243902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.243926 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:46:58.244061 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.244065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.244141 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.244266 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:58.245903 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.246767 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.249072 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.249940 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.251553 (+ 1613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.255329 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:58.255339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:58.255347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:58.255394 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:58.259275 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.260102 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.260107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.260110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.260228 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.260783 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.260791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.260794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.261180 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:58.261336 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.262072 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.262302 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7518,"thread_start_us":7415,"threads_started":1}
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:46:58.263923 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.233571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:39243 (local address 127.31.26.194:36931)
0927 18:46:58.239719 (+ 6148us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.239769 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.242934 (+ 3165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.244549 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.244559 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.245019 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:58.245718 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.245737 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.246960 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.246966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.248917 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.248943 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.250438 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.250447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.250866 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.252082 (+ 1216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:58.252116 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:58.255031 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:58.259450 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:58.259467 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:58.259488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:58.259953 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:58.260352 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:58.260357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:58.260360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:58.260633 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:58.261318 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:58.261326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:58.261688 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.261976 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.262554 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5825,"thread_start_us":5584,"threads_started":1}
I20260927 18:46:58.265637 11086 heartbeater.cc:347] Connected to a master server at 127.31.26.254:39243
I20260927 18:46:58.265969 11086 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:46:58.266851 11086 heartbeater.cc:511] Master 127.31.26.254:39243 requested a full tablet report, sending...
I20260927 18:46:58.268669 10730 ts_manager.cc:194] Registered new tserver with Master: 3beccb6d607c4886b45cee96f01f91fb (127.31.26.194:37163)
I20260927 18:46:58.269791 10730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:36931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:46:58.300895 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:48761/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39243
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:46:58.453181 10943 heartbeater.cc:503] Master 127.31.26.254:39243 was elected leader, sending a full tablet report...
W20260927 18:46:58.678555 11092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:58.678848 11092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:58.679028 11092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:58.687018 11092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:58.687137 11092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:58.687350 11092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:46:58.697568 11092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.26.212:48761/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:39243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:46:58.698921 11092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:58.700321 11092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:58.715189 11100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:58.715214 11098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:58.715189 11097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:58.716403 11092 server_base.cc:1034] running on GCE node
I20260927 18:46:58.717211 11092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:58.718284 11092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:58.719537 11092 hybrid_clock.cc:648] HybridClock initialized: now 1790534818719479 us; error 53 us; skew 500 ppm
I20260927 18:46:58.719916 11092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:46:58 dist-test-slave-rfd5 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:58.723341 11092 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:46:58.725050 11092 webserver.cc:460] Webserver started at http://127.31.26.195:35165/ using document root <none> and password file <none>
I20260927 18:46:58.725873 11092 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:58.726038 11092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:58.726361 11092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:46:58 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:46:58.743774 11092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "6ee9301f5e914d858fdbc325ecdd6b5e"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-rfd5"
server_key: "42d834df33d016f70bc38451bc8e6ef0"
server_key_iv: "f21c16a066e6bf2aaf1530cea50e69e7"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:58.745025 11092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "6ee9301f5e914d858fdbc325ecdd6b5e"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-rfd5"
server_key: "42d834df33d016f70bc38451bc8e6ef0"
server_key_iv: "f21c16a066e6bf2aaf1530cea50e69e7"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:58.751996 11092 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:46:58.769356 11107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:58.771230 11092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:46:58.771443 11092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "6ee9301f5e914d858fdbc325ecdd6b5e"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-rfd5"
server_key: "42d834df33d016f70bc38451bc8e6ef0"
server_key_iv: "f21c16a066e6bf2aaf1530cea50e69e7"
server_key_version: "kuduclusterkey@0"
I20260927 18:46:58.771720 11092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:58.783984 11092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:58.788489 11092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:58.788828 11092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:58.789870 11092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:58.791431 11092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:58.791540 11092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:58.791684 11092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:58.791752 11092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:58.828274 11092 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:38929
I20260927 18:46:58.828313 11219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:38929 every 8 connection(s)
I20260927 18:46:58.829703 11092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:58.830294 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 11092
I20260927 18:46:58.830482 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:58.831142 31851 external_mini_cluster.cc:1428] Setting key 68f21ef519fa3cdd21e9ae7b96a444da
Sep 27 18:46:58 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:46:58.850816 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.832748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:43915 (local address 127.31.26.254:39243)
0927 18:46:58.833340 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.833345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.834544 (+ 1199us) server_negotiation.cc:407] Connection header received
0927 18:46:58.836776 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.836780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.836847 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.836940 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:58.838466 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.839188 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.840807 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.841316 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.841413 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.844837 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:58.844845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:58.844849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:58.844889 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:58.847821 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.848538 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.848541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.848544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.848665 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.849029 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.849032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.849034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.849359 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:58.849547 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.850341 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.850529 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":231,"threads_started":1}
I20260927 18:46:58.851873 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.833529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:39243 (local address 127.31.26.195:43915)
0927 18:46:58.834364 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.834423 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.836500 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.837233 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.837243 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.837708 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:58.838307 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.838324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.839340 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.839344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.840663 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.840683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.841525 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.841530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.841741 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.842479 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:58.842509 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:58.844563 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:58.847954 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:58.847962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:58.847977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:58.848427 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:58.848758 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:58.848761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:58.848763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:58.848935 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:58.849466 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:58.849472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:58.849916 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.850260 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.850727 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":338,"threads_started":1}
I20260927 18:46:58.853480 11220 heartbeater.cc:347] Connected to a master server at 127.31.26.254:39243
I20260927 18:46:58.853780 11220 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:58.854573 11220 heartbeater.cc:511] Master 127.31.26.254:39243 requested a full tablet report, sending...
I20260927 18:46:58.856045 10730 ts_manager.cc:194] Registered new tserver with Master: 6ee9301f5e914d858fdbc325ecdd6b5e (127.31.26.195:38929)
I20260927 18:46:58.856840 10730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:43915
I20260927 18:46:58.859645 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:58.878566 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.864403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.31.26.254:39243)
0927 18:46:58.864586 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.864590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.865021 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:46:58.865258 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.865262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.865344 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.865437 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:58.867287 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.868030 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.869514 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.869945 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.870059 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.872619 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:58.872626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:58.872629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:58.872666 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:58.875737 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.876657 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.876663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.876666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.876826 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.877346 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.877350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.877352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.877655 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:58.877839 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.878126 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.878308 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:46:58.882874 10730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:58.891258 10730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:58.916494 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.906717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:38929 (local address 127.0.0.1:52408)
0927 18:46:58.907386 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.907405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.907497 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.909042 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.909050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.909079 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:58.909522 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.909537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.911669 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.911677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.913341 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.913357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.915499 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.915507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.915712 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.915763 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.915914 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.916106 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":1654,"threads_started":1}
I20260927 18:46:58.916908 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.907054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.31.26.195:38929)
0927 18:46:58.908303 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.908315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.908358 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:46:58.908465 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.908474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.908818 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.909042 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:58.909750 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.911252 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.913503 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.915035 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.915178 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.915410 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.916369 (+ 959us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.916538 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":850,"threads_started":1}
I20260927 18:46:58.917784 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.905951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:35135 (local address 127.0.0.1:42324)
0927 18:46:58.907102 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.907131 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.907266 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.910402 (+ 3136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.910407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.910433 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:58.910803 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.910819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.913264 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.913271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.915224 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.915238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.917232 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.917237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.917357 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.917380 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.917431 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.917520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":4,"thread_start_us":861,"threads_started":1}
I20260927 18:46:58.919013 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.908015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.31.26.193:35135)
0927 18:46:58.909796 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.909806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.909837 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:46:58.909930 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.909940 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.910202 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.910393 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:58.911437 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.913066 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.915404 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.916973 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.918184 (+ 1211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.918425 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.918496 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.918682 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1610,"thread_start_us":1434,"threads_started":1}
I20260927 18:46:58.919052 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.905242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37163 (local address 127.0.0.1:41678)
0927 18:46:58.906061 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.906119 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.906322 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.910983 (+ 4661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.910988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.911011 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:58.911422 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.911433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.913429 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.913437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.915633 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.915644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.918235 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.918242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.918410 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.918442 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.918548 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.918675 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":2,"thread_start_us":341,"threads_started":1}
I20260927 18:46:58.919739 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.905438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.31.26.194:37163)
0927 18:46:58.910158 (+ 4720us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.910165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.910187 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:46:58.910258 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.910262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.910765 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.911625 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:58.911837 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.913294 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.915776 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.918007 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.918174 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.918377 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.919380 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.919526 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4555,"thread_start_us":4451,"threads_started":1}
I20260927 18:46:58.921209 11155 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f (DEFAULT_TABLE table=test-table [id=9c846faf84af4f859f46c2045ce293f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:58.922837 11155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:58.922406 10878 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f (DEFAULT_TABLE table=test-table [id=9c846faf84af4f859f46c2045ce293f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:58.923563 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:58.923607 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f (DEFAULT_TABLE table=test-table [id=9c846faf84af4f859f46c2045ce293f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:58.924840 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:58.934950 11239 tablet_bootstrap.cc:492] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Bootstrap starting.
I20260927 18:46:58.937912 11240 tablet_bootstrap.cc:492] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Bootstrap starting.
I20260927 18:46:58.939599 11239 tablet_bootstrap.cc:654] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:58.941112 11239 log.cc:824] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:58.941488 11240 tablet_bootstrap.cc:654] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:58.941819 11241 tablet_bootstrap.cc:492] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Bootstrap starting.
I20260927 18:46:58.942912 11240 log.cc:824] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:58.945698 11239 tablet_bootstrap.cc:492] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: No bootstrap required, opened a new log
I20260927 18:46:58.945878 11241 tablet_bootstrap.cc:654] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:58.946015 11240 tablet_bootstrap.cc:492] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: No bootstrap required, opened a new log
I20260927 18:46:58.946008 11239 ts_tablet_manager.cc:1397] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 18:46:58.946287 11240 ts_tablet_manager.cc:1397] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 18:46:58.947119 11241 log.cc:824] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:58.951704 11241 tablet_bootstrap.cc:492] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: No bootstrap required, opened a new log
I20260927 18:46:58.952039 11241 ts_tablet_manager.cc:1397] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 18:46:58.954839 11239 raft_consensus.cc:348] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.955237 11239 raft_consensus.cc:374] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:58.955338 11239 raft_consensus.cc:729] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf3594b63574342a2c7504d7c57d46b, State: Initialized, Role: FOLLOWER
I20260927 18:46:58.956001 11239 consensus_queue.cc:260] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.957479 11240 raft_consensus.cc:348] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.957850 11239 ts_tablet_manager.cc:1428] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 18:46:58.957899 11240 raft_consensus.cc:374] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:58.957984 11240 raft_consensus.cc:729] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee9301f5e914d858fdbc325ecdd6b5e, State: Initialized, Role: FOLLOWER
I20260927 18:46:58.958591 11240 consensus_queue.cc:260] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.959086 11246 raft_consensus.cc:482] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:58.959335 11246 raft_consensus.cc:504] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.961395 11246 leader_election.cc:288] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3beccb6d607c4886b45cee96f01f91fb (127.31.26.194:37163), 6ee9301f5e914d858fdbc325ecdd6b5e (127.31.26.195:38929)
I20260927 18:46:58.965126 11241 raft_consensus.cc:348] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.965791 11241 raft_consensus.cc:374] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:58.965914 11241 raft_consensus.cc:729] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3beccb6d607c4886b45cee96f01f91fb, State: Initialized, Role: FOLLOWER
I20260927 18:46:58.966887 11241 consensus_queue.cc:260] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.968211 11220 heartbeater.cc:503] Master 127.31.26.254:39243 was elected leader, sending a full tablet report...
I20260927 18:46:58.968989 11240 ts_tablet_manager.cc:1428] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Time spent starting tablet: real 0.023s user 0.016s sys 0.007s
I20260927 18:46:58.974395 11086 heartbeater.cc:503] Master 127.31.26.254:39243 was elected leader, sending a full tablet report...
I20260927 18:46:58.974807 11241 ts_tablet_manager.cc:1428] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Time spent starting tablet: real 0.023s user 0.007s sys 0.011s
I20260927 18:46:58.976713 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.962568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37163 (local address 127.31.26.193:36767)
0927 18:46:58.965705 (+ 3137us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.965728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.965951 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.966593 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.966597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.966627 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:58.967053 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.967066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.968831 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.968840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.972055 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.972079 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.975577 (+ 3498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.975586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.975909 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.975955 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.976074 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.976220 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2985,"thread_start_us":3987,"threads_started":1}
I20260927 18:46:58.976787 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.962547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:36767 (local address 127.31.26.194:37163)
0927 18:46:58.962778 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.962792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.965822 (+ 3030us) server_negotiation.cc:407] Connection header received
0927 18:46:58.966142 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.966149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.966261 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.966337 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:58.967262 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.968560 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.972332 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.974832 (+ 2500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.975005 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.975137 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.976398 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.976536 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:46:58.978453 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9140d3dd4d4077bfc7a42b3f5de24f" candidate_uuid: "4cf3594b63574342a2c7504d7c57d46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3beccb6d607c4886b45cee96f01f91fb" is_pre_election: true
I20260927 18:46:58.979084 11041 raft_consensus.cc:2454] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf3594b63574342a2c7504d7c57d46b in term 0.
I20260927 18:46:58.979897 10831 leader_election.cc:302] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3beccb6d607c4886b45cee96f01f91fb, 4cf3594b63574342a2c7504d7c57d46b; no voters:
I20260927 18:46:58.980438 11245 raft_consensus.cc:2781] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:58.980651 11245 raft_consensus.cc:482] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:58.980832 11245 raft_consensus.cc:3037] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:58.983825 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.967082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:38929 (local address 127.31.26.193:37671)
0927 18:46:58.975815 (+ 8733us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:58.975836 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:58.975971 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:58.977662 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:58.977668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:58.977699 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:58.978044 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.978067 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.979496 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.979501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.981533 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:58.981547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.983246 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.983250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:58.983378 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.983403 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:46:58.983483 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:46:58.983583 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8572,"thread_start_us":8403,"threads_started":1}
I20260927 18:46:58.983978 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.967481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:37671 (local address 127.31.26.195:38929)
0927 18:46:58.967713 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.967723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.975940 (+ 8217us) server_negotiation.cc:407] Connection header received
0927 18:46:58.976119 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.976124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.976202 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.976292 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:58.978246 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.979317 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.981714 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.983003 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.983185 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.983267 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.983565 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.983709 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:46:58.984673 11245 raft_consensus.cc:504] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.984875 11175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9140d3dd4d4077bfc7a42b3f5de24f" candidate_uuid: "4cf3594b63574342a2c7504d7c57d46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" is_pre_election: true
I20260927 18:46:58.985503 11245 leader_election.cc:288] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [CANDIDATE]: Term 1 election: Requested vote from peers 3beccb6d607c4886b45cee96f01f91fb (127.31.26.194:37163), 6ee9301f5e914d858fdbc325ecdd6b5e (127.31.26.195:38929)
I20260927 18:46:58.985528 11175 raft_consensus.cc:2454] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf3594b63574342a2c7504d7c57d46b in term 0.
I20260927 18:46:58.985975 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9140d3dd4d4077bfc7a42b3f5de24f" candidate_uuid: "4cf3594b63574342a2c7504d7c57d46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3beccb6d607c4886b45cee96f01f91fb"
I20260927 18:46:58.986215 11041 raft_consensus.cc:3037] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:58.986155 11174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9140d3dd4d4077bfc7a42b3f5de24f" candidate_uuid: "4cf3594b63574342a2c7504d7c57d46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e"
I20260927 18:46:58.986433 11174 raft_consensus.cc:3037] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:46:58.988024 11087 tablet.cc:2369] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:58.990193 11174 raft_consensus.cc:2454] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf3594b63574342a2c7504d7c57d46b in term 1.
I20260927 18:46:58.990224 11041 raft_consensus.cc:2454] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf3594b63574342a2c7504d7c57d46b in term 1.
I20260927 18:46:58.990638 10831 leader_election.cc:302] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3beccb6d607c4886b45cee96f01f91fb, 4cf3594b63574342a2c7504d7c57d46b; no voters:
I20260927 18:46:58.991014 11245 raft_consensus.cc:2781] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:58.991231 11245 raft_consensus.cc:686] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 1 LEADER]: Becoming Leader. State: Replica: 4cf3594b63574342a2c7504d7c57d46b, State: Running, Role: LEADER
I20260927 18:46:58.992165 11245 consensus_queue.cc:237] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } }
I20260927 18:46:58.997511 10729 catalog_manager.cc:5495] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cf3594b63574342a2c7504d7c57d46b (127.31.26.193). New cstate: current_term: 1 leader_uuid: "4cf3594b63574342a2c7504d7c57d46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cf3594b63574342a2c7504d7c57d46b" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35135 } health_report { overall_health: HEALTHY } } }
I20260927 18:46:59.026755 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.020754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.31.26.193:35135)
0927 18:46:59.021070 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:59.021076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:59.021123 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:46:59.021252 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:59.021256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:59.021343 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:59.021501 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:59.021891 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.023020 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.024473 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.024889 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.025026 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.025578 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:59.026102 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:59.026273 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:59.026387 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 18:46:59.026473 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260927 18:46:59.035190 11174 raft_consensus.cc:1261] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 1 FOLLOWER]: Refusing update from remote peer 4cf3594b63574342a2c7504d7c57d46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:59.035190 11041 raft_consensus.cc:1261] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 1 FOLLOWER]: Refusing update from remote peer 4cf3594b63574342a2c7504d7c57d46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:59.036180 11246 consensus_queue.cc:1035] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3beccb6d607c4886b45cee96f01f91fb" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:59.036510 11245 consensus_queue.cc:1035] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee9301f5e914d858fdbc325ecdd6b5e" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:59.066406 10730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:59.066762 10730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:59.070588 10730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96957facd6fb4b7a992574bf56ca8e65: Sending DeleteTablet for 3 replicas of tablet 4a9140d3dd4d4077bfc7a42b3f5de24f
I20260927 18:46:59.071903 11155 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:59 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:52408
I20260927 18:46:59.071980 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:59 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:42324
I20260927 18:46:59.072152 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a9140d3dd4d4077bfc7a42b3f5de24f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:59 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:41678
I20260927 18:46:59.075138 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 10810
I20260927 18:46:59.088682 11265 tablet_replica.cc:330] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: stopping tablet replica
I20260927 18:46:59.088979 11263 tablet_replica.cc:330] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: stopping tablet replica
I20260927 18:46:59.089228 11265 raft_consensus.cc:2229] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:59.089452 11263 raft_consensus.cc:2229] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:59.089789 11265 raft_consensus.cc:2258] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:59.089922 11263 raft_consensus.cc:2258] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:59.091744 11263 ts_tablet_manager.cc:1905] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:59.092432 11265 ts_tablet_manager.cc:1905] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:59.095674 11264 tablet_replica.cc:330] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: stopping tablet replica
I20260927 18:46:59.096319 11264 raft_consensus.cc:2229] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:59.097247 11264 raft_consensus.cc:2258] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:59.097478 11263 ts_tablet_manager.cc:1918] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:59.097640 11263 log.cc:1196] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/4a9140d3dd4d4077bfc7a42b3f5de24f
I20260927 18:46:59.098193 11263 ts_tablet_manager.cc:1939] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 6ee9301f5e914d858fdbc325ecdd6b5e: Deleting consensus metadata
I20260927 18:46:59.099936 11265 ts_tablet_manager.cc:1918] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:59.100122 11265 log.cc:1196] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/4a9140d3dd4d4077bfc7a42b3f5de24f
I20260927 18:46:59.100114 11264 ts_tablet_manager.cc:1905] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:59.100142 10714 catalog_manager.cc:4841] TS 6ee9301f5e914d858fdbc325ecdd6b5e (127.31.26.195:38929): tablet 4a9140d3dd4d4077bfc7a42b3f5de24f (table test-table [id=9c846faf84af4f859f46c2045ce293f2]) successfully deleted
I20260927 18:46:59.100740 11265 ts_tablet_manager.cc:1939] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 3beccb6d607c4886b45cee96f01f91fb: Deleting consensus metadata
I20260927 18:46:59.102968 10714 catalog_manager.cc:4841] TS 3beccb6d607c4886b45cee96f01f91fb (127.31.26.194:37163): tablet 4a9140d3dd4d4077bfc7a42b3f5de24f (table test-table [id=9c846faf84af4f859f46c2045ce293f2]) successfully deleted
Sep 27 18:46:59 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:46:59.107653 11264 ts_tablet_manager.cc:1918] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:59.107838 11264 log.cc:1196] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/4a9140d3dd4d4077bfc7a42b3f5de24f
I20260927 18:46:59.108973 11264 ts_tablet_manager.cc:1939] T 4a9140d3dd4d4077bfc7a42b3f5de24f P 4cf3594b63574342a2c7504d7c57d46b: Deleting consensus metadata
I20260927 18:46:59.111073 10715 catalog_manager.cc:4841] TS 4cf3594b63574342a2c7504d7c57d46b (127.31.26.193:35135): tablet 4a9140d3dd4d4077bfc7a42b3f5de24f (table test-table [id=9c846faf84af4f859f46c2045ce293f2]) successfully deleted
I20260927 18:46:59.115450 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.076350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.31.26.193:35135)
0927 18:46:59.076528 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:59.076536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:59.081611 (+ 5075us) server_negotiation.cc:407] Connection header received
0927 18:46:59.081926 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:59.081932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:59.082021 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:59.082103 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:59.084327 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.085928 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.100656 (+ 14728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.101496 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.101619 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.107677 (+ 6058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:59.107712 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:59.107733 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:59.107789 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:59.112317 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.113312 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.113320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.113322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.113470 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.113945 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.113949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.113951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.114314 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:59.114520 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:59.114943 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:46:59.115131 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:46:59.116267 10938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:59.217701 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 10810
I20260927 18:46:59.236354 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 10949
Sep 27 18:46:59 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:46:59.253430 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.237369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.31.26.194:37163)
0927 18:46:59.237579 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:59.237585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:59.237611 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:46:59.237671 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:59.237677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:59.237742 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:59.237843 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:59.239249 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.240408 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.241941 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.242592 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.242710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.247260 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:59.247274 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:59.247292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:59.247339 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:59.250544 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.251348 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.251356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.251358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.251552 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.251931 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.251934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.251936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.252325 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:59.252528 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:59.252820 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 18:46:59.252984 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:46:59.254158 11081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:59.344349 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 10949
I20260927 18:46:59.361639 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 11092
Sep 27 18:46:59 dist-test-slave-rfd5 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:46:59.378098 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.362399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.31.26.195:38929)
0927 18:46:59.362570 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:59.362574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:59.362592 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:46:59.362730 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:59.362734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:59.362794 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:59.362880 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:59.364330 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.365213 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.366539 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.366975 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.367149 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.371816 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:59.371835 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:59.371851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:59.371892 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:59.375102 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.375920 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.375931 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.375936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.376096 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.376513 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.376518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.376520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.377143 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:59.377372 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:59.377659 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 18:46:59.377833 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:46:59.378703 11215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:59.468830 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 11092
I20260927 18:46:59.485865 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 10698
I20260927 18:46:59.487159 10760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:46:59.604694 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 10698
I20260927 18:46:59.628574 31851 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:46:59.964756 31851 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:46:59.964959 31851 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:47:00.310757 31851 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:47:00.311 UTC [10284] LOG: received smart shutdown request
2026-09-27 18:47:00.314 UTC [10284] LOG: background worker "logical replication launcher" (PID 10293) exited with exit code 1
2026-09-27 18:47:00.315 UTC [10288] LOG: shutting down
2026-09-27 18:47:00.495 UTC [10284] LOG: database system is shut down
2026-09-27T18:47:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51594 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:00 dist-test-slave-rfd5 krb5kdc[11279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:00 dist-test-slave-rfd5 krb5kdc[11279](info): set up 2 sockets
Sep 27 18:47:00 dist-test-slave-rfd5 krb5kdc[11279](info): commencing operation
krb5kdc: starting...
W20260927 18:47:01.741600 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:47:01 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/rangeradmin_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/rangeradmin_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/rangerlookup_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/rangerlookup_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
I20260927 18:47:02.072659 31851 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskk9LG2U/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/postgres -l logfile start
2026-09-27 18:47:04.694 UTC [11309] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:47:04.694 UTC [11309] LOG: listening on IPv4 address "127.31.26.212", port 37405
2026-09-27 18:47:04.698 UTC [11309] LOG: listening on Unix socket "/tmp/.s.PGSQL.37405"
2026-09-27 18:47:04.721 UTC [11312] LOG: database system was shut down at 2026-09-27 18:47:03 UTC
2026-09-27 18:47:04.725 UTC [11309] LOG: database system is ready to accept connections
I20260927 18:47:05.697233 31851 mini_postgres.cc:96] Postgres bound to 37405
2026-09-27 18:47:05.714 UTC [11320] FATAL: database "slave" does not exist
127.31.26.212:37405 - accepting connections
I20260927 18:47:05.714845 31851 mini_ranger.cc:161] Starting Ranger...
I20260927 18:47:05.742064 31851 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:47:06.259518 31851 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:47:06.259624 31851 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-admin
2026-09-27 18:47:06,560 [I] DB FLAVOR :POSTGRES
2026-09-27 18:47:06,561 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:47:06,561 [I] Checking connection..
2026-09-27 18:47:06,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:47:06,835 [I] Checking connection passed.
2026-09-27 18:47:06,835 [I] --------- Verifying version history table ---------
2026-09-27 18:47:06,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:47:07,109 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:47:07,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:47:07,355 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:47:07,356 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:47:07,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:47:07.622 UTC [11384] WARNING: there is no transaction in progress
2026-09-27 18:47:07,628 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:47:07,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:47:07,888 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:47:07,888 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:47:07,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:47:08,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:47:08,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rfd5.c.gcp-upstream.internal','N') ;"
2026-09-27 18:47:08,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:47:08,948 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:47:08,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:47:09,217 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:47:09,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:47:09,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:47:09,729 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:47:09,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:47:10.773 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.792 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.812 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.830 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.847 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.870 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.932 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.940 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.956 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.967 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.979 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:10.994 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:11.005 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:11.012 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:11.020 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:11.028 UTC [11492] WARNING: there is no transaction in progress
2026-09-27 18:47:11,767 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:47:11,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:47:12,003 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:47:12,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:47:12,248 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:47:12,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:47:12,509 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:47:12,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:47:12,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:47:13,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rfd5.c.gcp-upstream.internal';"
2026-09-27 18:47:13,264 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:47:13,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:47:13,511 [I] DB_PATCHES have already been applied
I20260927 18:47:13.517378 31851 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-admin:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskk9LG2U/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:47:13.517477 31851 mini_ranger.cc:193] Using host: 127.31.26.212
I20260927 18:47:13.532673 31851 mini_ranger.cc:227] Ranger admin URL: http://127.31.26.212:43417
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:47:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:47:14.707764 31851 mini_ranger.cc:160] Time spent starting Ranger: real 8.993s user 0.000s sys 0.041s
Sep 27, 2026 6:47:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:47:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 6:47:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:47:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:47:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:47:37.987991 31851 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.26.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.26.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/rangerkms_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.26.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/rangerkms_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.26.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.26.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.26.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/HTTP_127.31.26.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:47:38 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:47:38.172523 31851 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:47:38.196439 31851 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:47:39.167603 31851 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:47:39.167737 31851 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms
I20260927 18:47:39.167793 31851 mini_ranger_kms.cc:227] Using postgres at 127.31.26.212:37405
2026-09-27 18:47:39,536 [I] DB FLAVOR :POSTGRES
2026-09-27 18:47:39,536 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:47:39,536 [I] Checking connection
2026-09-27 18:47:39,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:47:39,794 [I] connection success
2026-09-27 18:47:39,794 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:47:39,794 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:47:39,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:47:40,085 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:47:40,085 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:47:40,085 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:47:40,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.26.212:37405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:47:40,369 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:47:40.919391 31851 mini_ranger_kms.cc:310] Created kms service
I20260927 18:47:41.132354 31851 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:47:41.179174 31851 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:47:41 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:41.602069 31851 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:47:41.602406 31851 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms:/tmp/dist-test-taskk9LG2U/build/asan/bin/postgresql.jar:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskk9LG2U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:47:41.602461 31851 mini_ranger_kms.cc:240] Using host: 127.31.26.212
I20260927 18:47:41.621069 31851 mini_ranger_kms.cc:276] Ranger KMS PID: 11673
I20260927 18:47:41.621335 31851 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.26.212:34781
Sep 27, 2026 6:47:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:47:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 18:47:43.928634 31851 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.756s user 0.080s sys 0.304s
I20260927 18:47:43.928916 31851 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:47:43.929147 31851 mini_ranger_kms.cc:401] 127.31.26.212:34781/kms/v1/keys
Sep 27 18:47:44 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534864, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.26.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 18:47:44 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534864, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.26.212@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
Sep 27 18:47:44 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534864, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.26.212@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
Sep 27 18:47:47 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:47:47.391204 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:36963
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:36963
--ranger_config_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:47.746199 11723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:47.746531 11723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:47.746622 11723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:47.754258 11723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:47.754395 11723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:47.754442 11723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:47.754492 11723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:47.754542 11723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:47.765251 11723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.26.212:34781/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:36963
--ranger_config_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:47.766587 11723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:47.768004 11723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:47.781565 11731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:47.781651 11729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:47.781622 11728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:47.782485 11723 server_base.cc:1034] running on GCE node
I20260927 18:47:47.783381 11723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:47.785079 11723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:47.786415 11723 hybrid_clock.cc:648] HybridClock initialized: now 1790534867786372 us; error 49 us; skew 500 ppm
I20260927 18:47:47.786861 11723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:47 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:47.797564 11723 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:47:47.799392 11723 webserver.cc:460] Webserver started at http://127.31.26.254:35187/ using document root <none> and password file <none>
I20260927 18:47:47.800165 11723 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:47.800326 11723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:47.800683 11723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:47:47 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:47.864015 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "a7f3e4dfa6cb48b396d68813b755cae2"
format_stamp: "Formatted at 2026-09-27 18:47:47 on dist-test-slave-rfd5"
server_key: "e41d494279084d7244631d96b5c9b3ee"
server_key_iv: "2c2dc610f71d1d9fa4305614be0b4059"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:47.865018 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "a7f3e4dfa6cb48b396d68813b755cae2"
format_stamp: "Formatted at 2026-09-27 18:47:47 on dist-test-slave-rfd5"
server_key: "e41d494279084d7244631d96b5c9b3ee"
server_key_iv: "2c2dc610f71d1d9fa4305614be0b4059"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:47.871291 11723 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:47:47.888798 11738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:47.890547 11723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:47:47.890758 11723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "a7f3e4dfa6cb48b396d68813b755cae2"
format_stamp: "Formatted at 2026-09-27 18:47:47 on dist-test-slave-rfd5"
server_key: "e41d494279084d7244631d96b5c9b3ee"
server_key_iv: "2c2dc610f71d1d9fa4305614be0b4059"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:47.891000 11723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:47.919616 11723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:47.924288 11723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:47.924665 11723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:47.945695 11723 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:36963
I20260927 18:47:47.945719 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:36963 every 8 connection(s)
I20260927 18:47:47.947250 11723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:47.952016 11790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:47.957814 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 11723
I20260927 18:47:47.958067 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:47:47.958895 31851 external_mini_cluster.cc:1428] Setting key ce376368532267586e4937bc9fe399c4
I20260927 18:47:47.965135 11790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Bootstrap starting.
I20260927 18:47:47.968698 11790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:47.969960 11790 log.cc:824] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:47.974355 11790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: No bootstrap required, opened a new log
Sep 27 18:47:47 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:47.983249 11790 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:47.983806 11790 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:47.983971 11790 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f3e4dfa6cb48b396d68813b755cae2, State: Initialized, Role: FOLLOWER
I20260927 18:47:47.984606 11790 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:47.984903 11790 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:47.985133 11790 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:47.985327 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:47.961338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.31.26.254:36963)
0927 18:47:47.962389 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:47.962406 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:47.962455 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:47:47.964439 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:47.964467 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:47.964976 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:47.965547 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:47.967097 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:47.968616 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:47.970170 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:47.971460 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:47.971704 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:47.976622 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:47.976631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:47.976648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:47.976696 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:47.980202 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:47.981070 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:47.981132 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:47.981144 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:47.981340 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:47.981814 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:47.981817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:47.981820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:47.982470 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:47.982805 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:47.983326 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 18:47:47.983841 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":511,"threads_started":1}
I20260927 18:47:47.985366 11790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:47.988260 11790 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:47.989060 11790 leader_election.cc:302] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f3e4dfa6cb48b396d68813b755cae2; no voters:
I20260927 18:47:47.989840 11790 leader_election.cc:288] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:47.989985 11795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:47.990732 11795 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 LEADER]: Becoming Leader. State: Replica: a7f3e4dfa6cb48b396d68813b755cae2, State: Running, Role: LEADER
I20260927 18:47:47.991338 11795 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:47.992141 11790 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:47.994639 11796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } } }
I20260927 18:47:47.995071 11796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:47.994588 11797 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f3e4dfa6cb48b396d68813b755cae2. Latest consensus state: current_term: 1 leader_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } } }
I20260927 18:47:47.995504 11797 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:47.996796 11804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:48.008651 11804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:48.009598 11790 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:47:48.028981 11804 catalog_manager.cc:1347] Generated new cluster ID: 1d9f4a1350ad4017ad5967f7da1b5247
I20260927 18:47:48.029201 11804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:48.043078 11804 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:48.045310 11804 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:48.056656 11804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Generated new TSK 0
I20260927 18:47:48.058076 11804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:47:48.813781 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:36963
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:47:49 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534869, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:47:49.264169 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:49.264554 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:49.264722 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:49.273537 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:49.273748 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:49.273957 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:47:49.286693 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.26.212:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:49.288357 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:49.290035 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:49.316236 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:49.316257 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:49.326134 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:49.326650 11835 server_base.cc:1034] running on GCE node
I20260927 18:47:49.327530 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:49.328830 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:49.330128 11835 hybrid_clock.cc:648] HybridClock initialized: now 1790534869330050 us; error 69 us; skew 500 ppm
I20260927 18:47:49.330639 11835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:49 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534869, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:49.334537 11835 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:47:49.342061 11835 webserver.cc:460] Webserver started at http://127.31.26.193:46651/ using document root <none> and password file <none>
I20260927 18:47:49.342972 11835 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:49.343161 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:49.343672 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:47:49 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534869, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:49.371109 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "61015ce760dc4e059b376b485aeca9b2"
format_stamp: "Formatted at 2026-09-27 18:47:49 on dist-test-slave-rfd5"
server_key: "611fc3e9fd6b0d64101da605476d6707"
server_key_iv: "17bfec1e3b7cf8ce9ce80f4b5576629f"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:49.373440 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "61015ce760dc4e059b376b485aeca9b2"
format_stamp: "Formatted at 2026-09-27 18:47:49 on dist-test-slave-rfd5"
server_key: "611fc3e9fd6b0d64101da605476d6707"
server_key_iv: "17bfec1e3b7cf8ce9ce80f4b5576629f"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:49.384825 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.004s
I20260927 18:47:49.425185 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:49.427409 11835 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.007s sys 0.004s
I20260927 18:47:49.427727 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "61015ce760dc4e059b376b485aeca9b2"
format_stamp: "Formatted at 2026-09-27 18:47:49 on dist-test-slave-rfd5"
server_key: "611fc3e9fd6b0d64101da605476d6707"
server_key_iv: "17bfec1e3b7cf8ce9ce80f4b5576629f"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:49.428004 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:49.444533 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:49.457255 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:49.457700 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:49.471324 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:49.473517 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:49.473634 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:49.473825 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:49.473897 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:49.574536 11835 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:36841
I20260927 18:47:49.574592 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:36841 every 8 connection(s)
I20260927 18:47:49.576436 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:49.583837 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 11835
I20260927 18:47:49.584085 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:49.585038 31851 external_mini_cluster.cc:1428] Setting key 4b35e9c3d741274e3a378c2f6d474d2d
Sep 27 18:47:49 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534869, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:49.624957 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:49.588881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:36963 (local address 127.31.26.193:60487)
0927 18:47:49.595658 (+ 6777us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:49.595721 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:49.600239 (+ 4518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:49.601908 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:49.601924 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:49.602785 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:49.603926 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:49.603969 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:49.605853 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:49.605860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:49.608292 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:49.608313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:49.609913 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:49.609922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:49.610295 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:49.611178 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:49.611216 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:49.614621 (+ 3405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:49.619120 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:49.619132 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:49.619155 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:49.619916 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:49.620508 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:49.620513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:49.620518 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:49.620837 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:49.621709 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:49.621719 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:49.622245 (+ 526us) client_negotiation.cc:769] Sending connection context
0927 18:47:49.622619 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 18:47:49.623299 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6409,"thread_start_us":6104,"threads_started":1}
I20260927 18:47:49.626207 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:49.589629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:60487 (local address 127.31.26.254:36963)
0927 18:47:49.596405 (+ 6776us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:49.596412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:49.596441 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:47:49.600991 (+ 4550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:49.600998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:49.601142 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:49.601279 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:49.604218 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:49.605576 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:49.608534 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:49.609313 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:49.609450 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:49.614998 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:49.615005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:49.615008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:49.615068 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:49.618882 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:49.620129 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:49.620139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:49.620143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:49.620338 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:49.621001 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:49.621009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:49.621016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:49.621458 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:49.621608 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:49.625429 (+ 3821us) server_negotiation.cc:299] Negotiation successful
0927 18:47:49.625733 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6644,"thread_start_us":15486,"threads_started":1}
I20260927 18:47:49.628448 11968 heartbeater.cc:347] Connected to a master server at 127.31.26.254:36963
I20260927 18:47:49.628800 11968 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:49.629711 11968 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:49.632505 11755 ts_manager.cc:194] Registered new tserver with Master: 61015ce760dc4e059b376b485aeca9b2 (127.31.26.193:36841)
Sep 27 18:47:49 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534869, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:49.634862 11755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:60487
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
W20260927 18:47:49.688645 31851 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.26.194: real 0.100s user 0.000s sys 0.014s
I20260927 18:47:49.689785 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:36963
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:47:49 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534869, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
W20260927 18:47:50.164944 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:50.165452 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:50.165622 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:50.176717 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:50.176959 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:50.177189 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:47:50.192584 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.26.212:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:50.194767 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:50.196928 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:50.251753 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:50.252372 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:50.254427 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:50.265702 11974 server_base.cc:1034] running on GCE node
I20260927 18:47:50.266826 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:50.276201 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:50.277617 11974 hybrid_clock.cc:648] HybridClock initialized: now 1790534870277530 us; error 92 us; skew 500 ppm
I20260927 18:47:50.278105 11974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:50 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:50.282938 11974 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:47:50.288516 11974 webserver.cc:460] Webserver started at http://127.31.26.194:34955/ using document root <none> and password file <none>
I20260927 18:47:50.289783 11974 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:50.289970 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:50.290357 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:47:50 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:50.325742 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "4160af8206304004a6a5268b7c0b13d7"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "b0f29c0ab9a1a19584a153ae31d82fa1"
server_key_iv: "ede11b62df20b18a933c4ec173cf2300"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:50.327121 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "4160af8206304004a6a5268b7c0b13d7"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "b0f29c0ab9a1a19584a153ae31d82fa1"
server_key_iv: "ede11b62df20b18a933c4ec173cf2300"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:50.341737 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.007s
I20260927 18:47:50.363089 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:50.364830 11974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:47:50.365145 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "4160af8206304004a6a5268b7c0b13d7"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "b0f29c0ab9a1a19584a153ae31d82fa1"
server_key_iv: "ede11b62df20b18a933c4ec173cf2300"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:50.365518 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:50.386159 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:50.390722 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:50.391067 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:50.392184 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:50.393865 11974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:50.393975 11974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:50.394157 11974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:50.394238 11974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:50.429395 11974 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:35857
I20260927 18:47:50.429415 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:35857 every 8 connection(s)
I20260927 18:47:50.430789 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:50.434062 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 11974
I20260927 18:47:50.434238 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:50.434813 31851 external_mini_cluster.cc:1428] Setting key 9ad8b620938b8bbfae8b79841bf2058b
Sep 27 18:47:50 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:50.452750 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:50.433745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:38979 (local address 127.31.26.254:36963)
0927 18:47:50.434287 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:50.434295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:50.435601 (+ 1306us) server_negotiation.cc:407] Connection header received
0927 18:47:50.437939 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:50.437945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:50.438052 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:50.438176 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:50.439700 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:50.440458 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:50.442147 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:50.442570 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:50.442716 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:50.446542 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:50.446547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:50.446551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:50.446583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:50.449447 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:50.450283 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:50.450287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:50.450289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:50.450415 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:50.450885 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:50.450892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:50.450900 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:50.451197 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:50.451360 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:50.452102 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 18:47:50.452347 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":284,"threads_started":1}
I20260927 18:47:50.453873 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:50.434512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:36963 (local address 127.31.26.194:38979)
0927 18:47:50.435344 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:50.435394 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:50.437643 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:50.438448 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:50.438457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:50.438907 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:50.439520 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:50.439540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:50.440618 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:50.440628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:50.441986 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:50.442006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:50.442812 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:50.442822 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:50.443163 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:50.443954 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:50.443990 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:50.446276 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:50.449627 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:50.449637 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:50.449658 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:50.450128 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:50.450537 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:50.450542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:50.450545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:50.450757 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:50.451333 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:50.451342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:50.451727 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 18:47:50.452011 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 18:47:50.452590 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":370,"threads_started":1}
I20260927 18:47:50.455492 12111 heartbeater.cc:347] Connected to a master server at 127.31.26.254:36963
I20260927 18:47:50.455870 12111 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:47:50.456799 12111 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:50.458571 11755 ts_manager.cc:194] Registered new tserver with Master: 4160af8206304004a6a5268b7c0b13d7 (127.31.26.194:35857)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:47:50.459555 11755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:38979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:47:50.488579 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:36963
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:50.639345 11968 heartbeater.cc:503] Master 127.31.26.254:36963 was elected leader, sending a full tablet report...
W20260927 18:47:50.840031 12117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:50.840346 12117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:50.840503 12117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:50.848098 12117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:50.848237 12117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:50.848454 12117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:47:50.858922 12117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.26.212:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:50.860314 12117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:50.861874 12117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:50.876197 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:50.876215 12125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:50.876653 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:50.877281 12117 server_base.cc:1034] running on GCE node
I20260927 18:47:50.877996 12117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:50.879186 12117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:50.880467 12117 hybrid_clock.cc:648] HybridClock initialized: now 1790534870880392 us; error 74 us; skew 500 ppm
I20260927 18:47:50.880860 12117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:50 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:50.884655 12117 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:47:50.886456 12117 webserver.cc:460] Webserver started at http://127.31.26.195:38081/ using document root <none> and password file <none>
I20260927 18:47:50.887226 12117 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:50.887410 12117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:50.887754 12117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:47:50 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:50.903776 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "e9558d2735354500836fa7d335ec20b0"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "db80d43bef9c19273d9392ea39e5fefc"
server_key_iv: "790dd7c39c99d79bb8369bb0b18dba86"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:50.904877 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "e9558d2735354500836fa7d335ec20b0"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "db80d43bef9c19273d9392ea39e5fefc"
server_key_iv: "790dd7c39c99d79bb8369bb0b18dba86"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:50.910929 12117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 18:47:50.926184 12132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:50.927919 12117 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:47:50.928122 12117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "e9558d2735354500836fa7d335ec20b0"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "db80d43bef9c19273d9392ea39e5fefc"
server_key_iv: "790dd7c39c99d79bb8369bb0b18dba86"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:50.928396 12117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:50.948545 12117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:50.953205 12117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:50.953536 12117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:50.954558 12117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:50.956169 12117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:50.956278 12117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:50.956439 12117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:50.956516 12117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:50.991664 12117 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:37193
I20260927 18:47:50.991703 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:37193 every 8 connection(s)
I20260927 18:47:50.993259 12117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:50.993952 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12117
I20260927 18:47:50.994160 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:50.994777 31851 external_mini_cluster.cc:1428] Setting key f1aafe11c5b6330d17b9b8c013cfd4d6
Sep 27 18:47:51 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:51.015810 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:50.996508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:37553 (local address 127.31.26.254:36963)
0927 18:47:50.997188 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:50.997195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:50.998301 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 18:47:51.000303 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:51.000308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:51.000402 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:51.000550 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:51.002275 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.003105 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.004934 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.005529 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.005685 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:51.009735 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:51.009741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:51.009745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:51.009781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:51.012721 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.013607 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.013613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.013616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.013751 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.014216 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.014219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.014221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.014507 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:51.014620 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:51.015341 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 18:47:51.015542 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":291,"threads_started":1}
I20260927 18:47:51.016815 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:50.997315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:36963 (local address 127.31.26.195:37553)
0927 18:47:50.998135 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:50.998182 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:51.000002 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:51.000828 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:51.000839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:51.001400 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:51.002072 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:51.002097 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.003291 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.003296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:51.004750 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:51.004764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.005792 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.005802 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:51.006094 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:51.007146 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:51.007173 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:51.009471 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:51.012873 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:51.012881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:51.012896 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:51.013478 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:51.013853 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:51.013856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:51.013858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:51.014095 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:51.014617 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:51.014624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:51.014991 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 18:47:51.015226 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 18:47:51.015635 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":340,"threads_started":1}
I20260927 18:47:51.018412 12245 heartbeater.cc:347] Connected to a master server at 127.31.26.254:36963
I20260927 18:47:51.018772 12245 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:51.019510 12245 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:51.021059 11755 ts_manager.cc:194] Registered new tserver with Master: e9558d2735354500836fa7d335ec20b0 (127.31.26.195:37193)
I20260927 18:47:51.022169 11755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:37553
I20260927 18:47:51.023970 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:47:51.025758 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 11835
Sep 27 18:47:51 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:47:51.044600 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:51.026793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.31.26.193:36841)
0927 18:47:51.027485 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:51.027492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:51.027521 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:47:51.027586 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:51.027593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:51.027830 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:51.027995 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:51.029515 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.030674 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.032706 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.033518 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.033637 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:51.038309 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:51.038325 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:51.038342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:51.038390 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:51.041598 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.042533 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.042541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.042544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.042724 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.043131 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.043134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.043136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.043556 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:51.043801 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:51.044087 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:47:51.044292 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":372,"threads_started":1}
I20260927 18:47:51.046135 11963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:51.131506 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 11835
I20260927 18:47:51.148046 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 11974
Sep 27 18:47:51 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:47:51.170291 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:51.149333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.31.26.194:35857)
0927 18:47:51.149831 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:51.149838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:51.149862 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:47:51.149920 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:51.149926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:51.150147 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:51.150321 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:51.151640 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.152648 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.154242 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.154837 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.154992 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:51.163835 (+ 8843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:51.163852 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:51.163869 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:51.163924 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:51.166949 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.167888 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.167899 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.167902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.168063 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.168477 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.168487 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.168489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.168877 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:51.169232 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:51.169642 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 18:47:51.169930 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260927 18:47:51.171862 12106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:51.256062 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 11974
I20260927 18:47:51.272751 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12117
Sep 27 18:47:51 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:47:51.291235 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:51.273864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.31.26.195:37193)
0927 18:47:51.274509 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:51.274517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:51.274548 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:47:51.274607 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:51.274614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:51.274837 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:51.274975 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:51.276320 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.277540 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.278903 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:51.279480 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:51.279590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:51.284589 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:51.284606 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:51.284624 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:51.284673 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:51.287994 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.288926 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.288935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.288937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.289273 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:51.289661 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:51.289665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:51.289667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:51.290048 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:51.290286 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:51.290598 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 18:47:51.290812 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":326,"threads_started":1}
I20260927 18:47:51.292712 12240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:51.376839 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12117
I20260927 18:47:51.392810 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 11723
I20260927 18:47:51.394342 11785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:51.502539 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 11723
I20260927 18:47:51.524176 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:36963
--webserver_interface=127.31.26.254
--webserver_port=35187
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:36963
--ranger_config_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:51.870224 12257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:51.870551 12257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:51.870642 12257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:51.878279 12257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:51.878412 12257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:51.878464 12257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:51.878513 12257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:51.878562 12257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:51.889225 12257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:36963
--ranger_config_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=35187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:51.890594 12257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:51.891965 12257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:51.904620 12262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:51.904644 12263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:51.904644 12265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:51.905480 12257 server_base.cc:1034] running on GCE node
I20260927 18:47:51.906387 12257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:51.907939 12257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:51.909214 12257 hybrid_clock.cc:648] HybridClock initialized: now 1790534871909140 us; error 71 us; skew 500 ppm
I20260927 18:47:51.909638 12257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:51 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534871, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:51.913203 12257 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:47:51.914865 12257 webserver.cc:460] Webserver started at http://127.31.26.254:35187/ using document root <none> and password file <none>
I20260927 18:47:51.915566 12257 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:51.915688 12257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:47:51 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534871, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:51.975229 12257 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:47:51.979115 12273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:51.980792 12257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:47:51.981051 12257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "a7f3e4dfa6cb48b396d68813b755cae2"
format_stamp: "Formatted at 2026-09-27 18:47:47 on dist-test-slave-rfd5"
server_key: "e41d494279084d7244631d96b5c9b3ee"
server_key_iv: "2c2dc610f71d1d9fa4305614be0b4059"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:51.982190 12257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:52.000396 12257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:52.005336 12257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:52.005703 12257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:52.027065 12257 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:36963
I20260927 18:47:52.027084 12324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:36963 every 8 connection(s)
I20260927 18:47:52.028957 12257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:52.036641 12325 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:47:52.039516 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12257
I20260927 18:47:52.040627 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:36841
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:36963
--webserver_port=46651
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:52.040854 12325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Bootstrap starting.
I20260927 18:47:52.062770 12325 log.cc:824] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:52.072932 12325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:47:52.073686 12325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Bootstrap complete.
I20260927 18:47:52.083097 12325 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:52.084014 12325 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f3e4dfa6cb48b396d68813b755cae2, State: Initialized, Role: FOLLOWER
I20260927 18:47:52.084749 12325 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:52.085024 12325 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:52.085175 12325 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:52.085395 12325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:47:52.090003 12325 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:52.090595 12325 leader_election.cc:302] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f3e4dfa6cb48b396d68813b755cae2; no voters:
I20260927 18:47:52.091298 12325 leader_election.cc:288] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:47:52.091452 12329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:47:52.093976 12325 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:52.094534 12329 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [term 2 LEADER]: Becoming Leader. State: Replica: a7f3e4dfa6cb48b396d68813b755cae2, State: Running, Role: LEADER
I20260927 18:47:52.095268 12329 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } }
I20260927 18:47:52.100692 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f3e4dfa6cb48b396d68813b755cae2. Latest consensus state: current_term: 2 leader_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } } }
I20260927 18:47:52.101235 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:52.101908 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f3e4dfa6cb48b396d68813b755cae2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 36963 } } }
I20260927 18:47:52.102190 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:52.102232 12342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:52.115872 12342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:52.118649 12342 catalog_manager.cc:1259] Loaded cluster ID: 1d9f4a1350ad4017ad5967f7da1b5247
I20260927 18:47:52.118878 12342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:52.123060 12342 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:52.124755 12342 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Loaded TSK: 0
I20260927 18:47:52.126566 12342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:52.129150 12325 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 18:47:52.533025 12327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:52.533636 12327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:52.533855 12327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:52.546953 12327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:52.547236 12327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:52.547502 12327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:47:52.564448 12327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:36841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=46651
--tserver_master_addrs=127.31.26.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:52.566706 12327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:52.568589 12327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:52.629467 12364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:52.630645 12363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:52.637416 12366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:52.638782 12327 server_base.cc:1034] running on GCE node
I20260927 18:47:52.639703 12327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:52.641837 12327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:52.643131 12327 hybrid_clock.cc:648] HybridClock initialized: now 1790534872643056 us; error 79 us; skew 500 ppm
I20260927 18:47:52.643523 12327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:52 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534872, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:52.652500 12327 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:47:52.667654 12327 webserver.cc:460] Webserver started at http://127.31.26.193:46651/ using document root <none> and password file <none>
I20260927 18:47:52.668526 12327 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:52.668712 12327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:47:52 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534872, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:52.701512 12327 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:47:52.707824 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:52.710026 12327 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260927 18:47:52.710242 12327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "61015ce760dc4e059b376b485aeca9b2"
format_stamp: "Formatted at 2026-09-27 18:47:49 on dist-test-slave-rfd5"
server_key: "611fc3e9fd6b0d64101da605476d6707"
server_key_iv: "17bfec1e3b7cf8ce9ce80f4b5576629f"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:52.711787 12327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:52.748153 12327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:52.753525 12327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:52.753842 12327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:52.755120 12327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:52.757103 12327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:52.757236 12327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:47:52.757401 12327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:52.757516 12327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:52.840034 12327 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:36841
I20260927 18:47:52.840128 12487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:36841 every 8 connection(s)
I20260927 18:47:52.841825 12327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:52.845304 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12327
I20260927 18:47:52.846187 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:35857
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:36963
--webserver_port=34955
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:47:52 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534872, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:52.899554 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:52.846086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:36963 (local address 127.31.26.193:51573)
0927 18:47:52.857335 (+ 11249us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:52.857406 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:52.860248 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:52.862233 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:52.862250 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:52.862845 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:52.863618 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:52.863644 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:52.865750 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:52.865757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:52.868136 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:52.868159 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:52.869872 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:52.869881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:52.870179 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:52.875467 (+ 5288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:52.875525 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:52.882201 (+ 6676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:52.887651 (+ 5450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:52.887662 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:52.887680 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:52.888443 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:52.892201 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:52.892207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:52.892211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:52.892630 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:52.896457 (+ 3827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:52.896469 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:52.896993 (+ 524us) client_negotiation.cc:769] Sending connection context
0927 18:47:52.897454 (+ 461us) client_negotiation.cc:240] Negotiation successful
0927 18:47:52.898100 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10830,"thread_start_us":10604,"threads_started":1}
I20260927 18:47:52.902350 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:52.845903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:51573 (local address 127.31.26.254:36963)
0927 18:47:52.860518 (+ 14615us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:52.860536 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:52.860588 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:47:52.860967 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:52.860982 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:52.861525 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:52.862010 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:52.863890 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:52.865485 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:52.868345 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:52.869153 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:52.869377 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:52.882686 (+ 13309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:52.882707 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:52.882735 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:52.882808 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:52.887373 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:52.888747 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:52.888757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:52.888767 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:52.889026 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:52.892841 (+ 3815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:52.892848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:52.892852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:52.893746 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:52.893984 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:52.900426 (+ 6442us) server_negotiation.cc:299] Negotiation successful
0927 18:47:52.900922 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14320,"thread_start_us":14066,"threads_started":1}
I20260927 18:47:52.907222 12488 heartbeater.cc:347] Connected to a master server at 127.31.26.254:36963
I20260927 18:47:52.907615 12488 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:52.908560 12488 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
W20260927 18:47:52.912989 12488 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:36963 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:47:52.913342 12488 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:52.913686 12488 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:52.915457 12488 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:52.915810 12488 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
W20260927 18:47:52.917418 12488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:47:53.328568 12492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:53.329195 12492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:53.329465 12492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:53.342042 12492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:53.342221 12492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:53.342479 12492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:47:53.359649 12492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:35857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=34955
--tserver_master_addrs=127.31.26.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:53.361537 12492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:53.363418 12492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:53.385637 12509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:53.386428 12508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 27 18:47:53 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534873, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:47:53.394822 12511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:53.396068 12492 server_base.cc:1034] running on GCE node
I20260927 18:47:53.396972 12492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:53.400343 12492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:53.401712 12492 hybrid_clock.cc:648] HybridClock initialized: now 1790534873401627 us; error 83 us; skew 500 ppm
I20260927 18:47:53.402222 12492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:53 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534873, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:53.406864 12492 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:47:53.424731 12492 webserver.cc:460] Webserver started at http://127.31.26.194:34955/ using document root <none> and password file <none>
I20260927 18:47:53.426015 12492 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:53.426281 12492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:47:53 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534873, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:53.463637 12492 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.010s sys 0.001s
I20260927 18:47:53.467808 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:53.469940 12492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:47:53.470304 12492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "4160af8206304004a6a5268b7c0b13d7"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "b0f29c0ab9a1a19584a153ae31d82fa1"
server_key_iv: "ede11b62df20b18a933c4ec173cf2300"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:53.471511 12492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:53.485395 12492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:53.491254 12492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:53.491616 12492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:53.492978 12492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:53.495051 12492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:53.495179 12492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:53.495296 12492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:53.495375 12492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:53.558501 12492 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:35857
I20260927 18:47:53.558533 12632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:35857 every 8 connection(s)
I20260927 18:47:53.560452 12492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:53.567549 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12492
I20260927 18:47:53.568334 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.26.212:34781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:37193
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:36963
--webserver_port=38081
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:47:53 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534873, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:53.621937 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:53.572583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:33855 (local address 127.31.26.254:36963)
0927 18:47:53.594538 (+ 21955us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:53.594547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:53.595754 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 18:47:53.599900 (+ 4146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:53.599907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:53.600023 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:53.600120 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:53.602903 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:53.604076 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:53.607031 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:53.607920 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:53.608062 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:53.613306 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:53.613318 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:53.613323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:53.613377 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:53.617299 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:53.618742 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:53.618749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:53.618753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:53.618929 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:53.619603 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:53.619608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:53.619611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:53.619998 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:53.620152 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:53.621248 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0927 18:47:53.621537 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21757,"thread_start_us":21987,"threads_started":1}
I20260927 18:47:53.624778 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:53.571876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:36963 (local address 127.31.26.194:33855)
0927 18:47:53.595339 (+ 23463us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:53.595410 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:53.598363 (+ 2953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:53.600776 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:53.600866 (+ 90us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:53.601630 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:53.602620 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:53.602654 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:53.604356 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:53.604361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:53.606777 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:53.606802 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:53.608558 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:53.608570 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:53.608922 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:53.609880 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:53.609923 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:53.612876 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:53.617685 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:53.617696 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:53.617718 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:53.618435 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:53.619116 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:53.619121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:53.619125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:53.619434 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:53.620251 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:53.620259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:53.620745 (+ 486us) client_negotiation.cc:769] Sending connection context
0927 18:47:53.622621 (+ 1876us) client_negotiation.cc:240] Negotiation successful
0927 18:47:53.623242 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23051,"thread_start_us":26792,"threads_started":1}
I20260927 18:47:53.627100 12633 heartbeater.cc:347] Connected to a master server at 127.31.26.254:36963
I20260927 18:47:53.627480 12633 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:53.628458 12633 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:53.630971 12290 ts_manager.cc:194] Registered new tserver with Master: 4160af8206304004a6a5268b7c0b13d7 (127.31.26.194:35857)
I20260927 18:47:53.633504 12290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:33855
Sep 27 18:47:53 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534873, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:53.917542 12488 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:53.918040 12488 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:53.919600 12290 ts_manager.cc:194] Registered new tserver with Master: 61015ce760dc4e059b376b485aeca9b2 (127.31.26.193:36841)
I20260927 18:47:53.920914 12290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:51573
W20260927 18:47:54.025580 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:54.026022 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:54.026242 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:54.038112 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:54.038295 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:54.038483 12637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:47:54.048921 12637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.26.212:34781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:37193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=38081
--tserver_master_addrs=127.31.26.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:54.050508 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:54.052016 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:54.072614 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:54.078001 12643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:54.078662 12645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:54.079207 12637 server_base.cc:1034] running on GCE node
I20260927 18:47:54.079968 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:54.081290 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:54.082574 12637 hybrid_clock.cc:648] HybridClock initialized: now 1790534874082509 us; error 86 us; skew 500 ppm
I20260927 18:47:54.082957 12637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:54 dist-test-slave-rfd5 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534874, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:54.086474 12637 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:47:54.090493 12637 webserver.cc:460] Webserver started at http://127.31.26.195:38081/ using document root <none> and password file <none>
I20260927 18:47:54.091305 12637 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:54.091475 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:47:54 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534874, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:54.124948 12637 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.005s sys 0.008s
I20260927 18:47:54.128481 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:54.130369 12637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:47:54.130602 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "e9558d2735354500836fa7d335ec20b0"
format_stamp: "Formatted at 2026-09-27 18:47:50 on dist-test-slave-rfd5"
server_key: "db80d43bef9c19273d9392ea39e5fefc"
server_key_iv: "790dd7c39c99d79bb8369bb0b18dba86"
server_key_version: "kuduclusterkey@0"
I20260927 18:47:54.131793 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:54.147883 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:54.152529 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:54.152904 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:54.154155 12637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:54.155923 12637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:54.156123 12637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:47:54.156284 12637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:54.156397 12637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:47:54 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534873, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for HTTP/127.31.26.212@KRBTEST.COM
I20260927 18:47:54.241973 12637 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:37193
I20260927 18:47:54.242010 12766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:37193 every 8 connection(s)
I20260927 18:47:54.243629 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:54.247282 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12637
Sep 27 18:47:54 dist-test-slave-rfd5 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534874, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:54.282253 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.248713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.31.26.254:36963)
0927 18:47:54.249877 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.249884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.260850 (+ 10966us) server_negotiation.cc:407] Connection header received
0927 18:47:54.260960 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.260970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.261152 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.261290 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:54.264746 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.266053 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.268575 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.269308 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.269539 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.273369 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:54.273376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:54.273381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:54.273444 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:54.277442 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.279695 (+ 2253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.279703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.279706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.279873 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.280356 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.280364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.280367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.280816 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:54.281082 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.281561 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.281846 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":2392,"threads_started":1}
I20260927 18:47:54.292073 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.254905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:40715 (local address 127.31.26.254:36963)
0927 18:47:54.261452 (+ 6547us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.261459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.261483 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:47:54.264121 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.264127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.264249 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.264377 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:54.266856 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.268399 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.271424 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.272168 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.272277 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.279762 (+ 7485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:54.279770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:54.279775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:54.279840 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:54.284427 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.288846 (+ 4419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.288855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.288862 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.289134 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.289859 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.289865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.289868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.290309 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:54.290477 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.291357 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.291631 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6400,"spinlock_wait_cycles":6912,"thread_start_us":6178,"threads_started":1}
I20260927 18:47:54.293197 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.254127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:36963 (local address 127.31.26.195:40715)
0927 18:47:54.260209 (+ 6082us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:54.260278 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:54.263675 (+ 3397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:54.264765 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:54.264781 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:54.265586 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:54.266583 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.266612 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.268674 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.268682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.271212 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.271233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.272907 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.272922 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.273342 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.274698 (+ 1356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:54.274741 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:54.279363 (+ 4622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:54.284654 (+ 5291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:54.284666 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:54.284688 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:54.285418 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:54.289286 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:54.289293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:54.289297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:54.289697 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:54.290450 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:54.290459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:54.290935 (+ 476us) client_negotiation.cc:769] Sending connection context
0927 18:47:54.291216 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 18:47:54.291677 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5688,"thread_start_us":5387,"threads_started":1}
I20260927 18:47:54.295158 12767 heartbeater.cc:347] Connected to a master server at 127.31.26.254:36963
I20260927 18:47:54.295570 12767 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:54.296546 12767 heartbeater.cc:511] Master 127.31.26.254:36963 requested a full tablet report, sending...
I20260927 18:47:54.298509 12290 ts_manager.cc:194] Registered new tserver with Master: e9558d2735354500836fa7d335ec20b0 (127.31.26.195:37193)
I20260927 18:47:54.299584 12290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:40715
I20260927 18:47:54.311409 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:47:54.360445 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.333890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.31.26.254:36963)
0927 18:47:54.334157 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.334164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.341201 (+ 7037us) server_negotiation.cc:407] Connection header received
0927 18:47:54.341874 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.341882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.341976 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.342060 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:54.344520 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.345726 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.347924 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.348657 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.348776 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.352950 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:54.352957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:54.352963 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:54.353018 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:54.357182 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.358235 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.358243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.358251 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.358399 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.358876 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.358881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.358883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.359192 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:54.359461 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.359828 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.360085 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:47:54.367602 12290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:54.378209 12290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:54.475986 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.418086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:37193 (local address 127.0.0.1:55112)
0927 18:47:54.456738 (+ 38652us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:54.456765 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:54.456952 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:54.458458 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:54.458470 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:54.458518 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:54.459076 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.459093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.463419 (+ 4326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.463430 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.472225 (+ 8795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.472239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.474842 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.474850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.475121 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.475193 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 18:47:54.475347 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 18:47:54.475574 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38474,"thread_start_us":37333,"threads_started":1}
I20260927 18:47:54.477005 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.418683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.31.26.195:37193)
0927 18:47:54.452991 (+ 34308us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.453006 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.457168 (+ 4162us) server_negotiation.cc:407] Connection header received
0927 18:47:54.457328 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.457340 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.457712 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.457990 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:54.459364 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.462233 (+ 2869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.472439 (+ 10206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.474162 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.474282 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.474579 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.476425 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.476640 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34011,"thread_start_us":34365,"threads_started":1}
I20260927 18:47:54.481293 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.463107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:35857 (local address 127.0.0.1:57626)
0927 18:47:54.467053 (+ 3946us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:54.467076 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:54.467218 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:54.471594 (+ 4376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:54.471598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:54.471630 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:54.472025 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.472042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.474112 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.474118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.476278 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.476299 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.478813 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.478820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.479010 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.479049 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:47:54.479139 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:47:54.480815 (+ 1676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3797,"mutex_wait_us":1,"thread_start_us":4176,"threads_started":1}
I20260927 18:47:54.482385 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.463342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.31.26.194:35857)
0927 18:47:54.471003 (+ 7661us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.471012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.471041 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:47:54.471147 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.471153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.471372 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.471597 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:54.472255 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.473888 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.476559 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.478269 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.478429 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.478644 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.481855 (+ 3211us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.482072 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7443,"thread_start_us":12953,"threads_started":1}
I20260927 18:47:54.484443 12702 tablet_service.cc:1467] Processing CreateTablet for tablet 7b15ce258ff742a0802b6d3ed33f8dfe (DEFAULT_TABLE table=test-table [id=39ad9d59ece84c708646bd2c2676c07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:54.486500 12702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b15ce258ff742a0802b6d3ed33f8dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:54.486507 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.416901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:36841 (local address 127.0.0.1:45728)
0927 18:47:54.454754 (+ 37853us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:54.454821 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:54.455083 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:54.470310 (+ 15227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:54.470315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:54.470347 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:54.470786 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.470803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.481162 (+ 10359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.481169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.483267 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.483287 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.485638 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.485648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.485850 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.485925 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 18:47:54.486008 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:47:54.486157 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37420,"thread_start_us":45518,"threads_started":1}
I20260927 18:47:54.487987 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.418565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.31.26.193:36841)
0927 18:47:54.469652 (+ 51087us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.469669 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.469699 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:47:54.469793 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.469799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.470043 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.470321 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:54.477708 (+ 7387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.479359 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.483740 (+ 4381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.485332 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.487022 (+ 1690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.487302 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.487389 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.487574 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50897,"thread_start_us":50729,"threads_started":1}
I20260927 18:47:54.487874 12568 tablet_service.cc:1467] Processing CreateTablet for tablet 7b15ce258ff742a0802b6d3ed33f8dfe (DEFAULT_TABLE table=test-table [id=39ad9d59ece84c708646bd2c2676c07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:54.489678 12568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b15ce258ff742a0802b6d3ed33f8dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:54.492267 12423 tablet_service.cc:1467] Processing CreateTablet for tablet 7b15ce258ff742a0802b6d3ed33f8dfe (DEFAULT_TABLE table=test-table [id=39ad9d59ece84c708646bd2c2676c07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:54.494273 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b15ce258ff742a0802b6d3ed33f8dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:54.517997 12789 tablet_bootstrap.cc:492] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Bootstrap starting.
I20260927 18:47:54.520284 12788 tablet_bootstrap.cc:492] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Bootstrap starting.
I20260927 18:47:54.526089 12788 tablet_bootstrap.cc:654] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:54.527158 12790 tablet_bootstrap.cc:492] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Bootstrap starting.
I20260927 18:47:54.529719 12789 tablet_bootstrap.cc:654] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:54.529670 12788 log.cc:824] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:54.531401 12789 log.cc:824] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:54.532085 12790 tablet_bootstrap.cc:654] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:54.534186 12790 log.cc:824] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:54.544672 12789 tablet_bootstrap.cc:492] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: No bootstrap required, opened a new log
I20260927 18:47:54.545070 12789 ts_tablet_manager.cc:1397] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20260927 18:47:54.555998 12788 tablet_bootstrap.cc:492] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: No bootstrap required, opened a new log
I20260927 18:47:54.556423 12788 ts_tablet_manager.cc:1397] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.004s
I20260927 18:47:54.556509 12790 tablet_bootstrap.cc:492] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: No bootstrap required, opened a new log
I20260927 18:47:54.556876 12790 ts_tablet_manager.cc:1397] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.015s
I20260927 18:47:54.561913 12789 raft_consensus.cc:348] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.562697 12789 raft_consensus.cc:374] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:54.562870 12789 raft_consensus.cc:729] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9558d2735354500836fa7d335ec20b0, State: Initialized, Role: FOLLOWER
I20260927 18:47:54.563820 12789 consensus_queue.cc:260] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.567565 12789 ts_tablet_manager.cc:1428] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Time spent starting tablet: real 0.022s user 0.009s sys 0.002s
I20260927 18:47:54.569109 12767 heartbeater.cc:503] Master 127.31.26.254:36963 was elected leader, sending a full tablet report...
I20260927 18:47:54.569003 12790 raft_consensus.cc:348] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.569578 12790 raft_consensus.cc:374] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:54.569680 12790 raft_consensus.cc:729] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61015ce760dc4e059b376b485aeca9b2, State: Initialized, Role: FOLLOWER
I20260927 18:47:54.570492 12790 consensus_queue.cc:260] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.576568 12488 heartbeater.cc:503] Master 127.31.26.254:36963 was elected leader, sending a full tablet report...
I20260927 18:47:54.577531 12790 ts_tablet_manager.cc:1428] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Time spent starting tablet: real 0.021s user 0.017s sys 0.002s
I20260927 18:47:54.584105 12788 raft_consensus.cc:348] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.584779 12788 raft_consensus.cc:374] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:54.584973 12788 raft_consensus.cc:729] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4160af8206304004a6a5268b7c0b13d7, State: Initialized, Role: FOLLOWER
I20260927 18:47:54.585891 12788 consensus_queue.cc:260] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.587443 12633 heartbeater.cc:503] Master 127.31.26.254:36963 was elected leader, sending a full tablet report...
I20260927 18:47:54.588130 12788 ts_tablet_manager.cc:1428] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Time spent starting tablet: real 0.032s user 0.011s sys 0.004s
W20260927 18:47:54.608018 12489 tablet.cc:2369] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:54.722028 12803 raft_consensus.cc:482] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:54.722415 12803 raft_consensus.cc:504] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.724251 12803 leader_election.cc:288] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61015ce760dc4e059b376b485aeca9b2 (127.31.26.193:36841), e9558d2735354500836fa7d335ec20b0 (127.31.26.195:37193)
I20260927 18:47:54.732577 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.725017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:60083 (local address 127.31.26.195:37193)
0927 18:47:54.725258 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.725264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.725717 (+ 453us) server_negotiation.cc:407] Connection header received
0927 18:47:54.726090 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.726098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.726188 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.726284 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:54.726900 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.727780 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.730348 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.731542 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.731734 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.731813 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.732169 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.732303 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:47:54.732700 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.724747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:37193 (local address 127.31.26.194:60083)
0927 18:47:54.725591 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:54.725615 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:54.725853 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:54.726313 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:54.726318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:54.726345 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:54.726722 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.726734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.727932 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.727937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.730114 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.730132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.731797 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.731803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.731995 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.732020 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:47:54.732138 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:47:54.732289 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":29,"thread_start_us":452,"threads_started":1}
I20260927 18:47:54.732782 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.724770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:58901 (local address 127.31.26.193:36841)
0927 18:47:54.724980 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.724987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.726217 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 18:47:54.726412 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.726419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.726511 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.726563 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:54.727358 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.728442 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.730805 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.731611 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.731811 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.731897 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.732401 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.732550 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:47:54.733563 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.724747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:36841 (local address 127.31.26.194:58901)
0927 18:47:54.726081 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:54.726100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:54.726232 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:54.726714 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:54.726717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:54.726735 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:54.727148 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.727159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.728616 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.728621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.730630 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:54.730640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.732016 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.732023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:54.732199 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.732233 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:47:54.733040 (+ 807us) client_negotiation.cc:240] Negotiation successful
0927 18:47:54.733205 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"mutex_wait_us":590,"thread_start_us":430,"threads_started":1}
I20260927 18:47:54.734095 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b15ce258ff742a0802b6d3ed33f8dfe" candidate_uuid: "4160af8206304004a6a5268b7c0b13d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9558d2735354500836fa7d335ec20b0" is_pre_election: true
I20260927 18:47:54.734733 12722 raft_consensus.cc:2454] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4160af8206304004a6a5268b7c0b13d7 in term 0.
I20260927 18:47:54.734900 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b15ce258ff742a0802b6d3ed33f8dfe" candidate_uuid: "4160af8206304004a6a5268b7c0b13d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61015ce760dc4e059b376b485aeca9b2" is_pre_election: true
I20260927 18:47:54.735435 12443 raft_consensus.cc:2454] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4160af8206304004a6a5268b7c0b13d7 in term 0.
I20260927 18:47:54.736310 12522 leader_election.cc:302] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4160af8206304004a6a5268b7c0b13d7, e9558d2735354500836fa7d335ec20b0; no voters:
I20260927 18:47:54.736852 12803 raft_consensus.cc:2781] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:54.737058 12803 raft_consensus.cc:482] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:54.737210 12803 raft_consensus.cc:3037] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:54.740840 12803 raft_consensus.cc:504] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.742164 12803 leader_election.cc:288] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [CANDIDATE]: Term 1 election: Requested vote from peers 61015ce760dc4e059b376b485aeca9b2 (127.31.26.193:36841), e9558d2735354500836fa7d335ec20b0 (127.31.26.195:37193)
I20260927 18:47:54.742449 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b15ce258ff742a0802b6d3ed33f8dfe" candidate_uuid: "4160af8206304004a6a5268b7c0b13d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61015ce760dc4e059b376b485aeca9b2"
I20260927 18:47:54.742481 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b15ce258ff742a0802b6d3ed33f8dfe" candidate_uuid: "4160af8206304004a6a5268b7c0b13d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9558d2735354500836fa7d335ec20b0"
I20260927 18:47:54.742693 12443 raft_consensus.cc:3037] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:54.742709 12722 raft_consensus.cc:3037] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:54.745036 12443 raft_consensus.cc:2454] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4160af8206304004a6a5268b7c0b13d7 in term 1.
I20260927 18:47:54.745558 12521 leader_election.cc:302] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4160af8206304004a6a5268b7c0b13d7, 61015ce760dc4e059b376b485aeca9b2; no voters:
I20260927 18:47:54.745878 12803 raft_consensus.cc:2781] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:54.745954 12722 raft_consensus.cc:2454] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4160af8206304004a6a5268b7c0b13d7 in term 1.
I20260927 18:47:54.746580 12803 raft_consensus.cc:686] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4160af8206304004a6a5268b7c0b13d7, State: Running, Role: LEADER
I20260927 18:47:54.747195 12803 consensus_queue.cc:237] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } }
I20260927 18:47:54.752904 12289 catalog_manager.cc:5495] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4160af8206304004a6a5268b7c0b13d7 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "4160af8206304004a6a5268b7c0b13d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4160af8206304004a6a5268b7c0b13d7" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 35857 } health_report { overall_health: HEALTHY } } }
W20260927 18:47:54.759816 12770 tablet.cc:2369] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:54.800221 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.793618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.31.26.194:35857)
0927 18:47:54.793858 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.793864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.793885 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:47:54.793990 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.793995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.794125 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.794280 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:54.794977 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.795857 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.797582 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.798158 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.798323 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.798961 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:54.799547 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:54.799747 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.799810 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.799904 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:47:54.809828 12722 raft_consensus.cc:1261] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 1 FOLLOWER]: Refusing update from remote peer 4160af8206304004a6a5268b7c0b13d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:54.809831 12443 raft_consensus.cc:1261] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 1 FOLLOWER]: Refusing update from remote peer 4160af8206304004a6a5268b7c0b13d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:54.811093 12806 consensus_queue.cc:1035] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9558d2735354500836fa7d335ec20b0" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:54.811583 12803 consensus_queue.cc:1035] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61015ce760dc4e059b376b485aeca9b2" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:47:54.819269 12634 tablet.cc:2369] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:54.830986 12809 mvcc.cc:205] Tried to move back new op lower bound from 7334030847208562688 to 7334030846969356288. Current Snapshot: MvccSnapshot[applied={T|T < 7334030847208562688}]
I20260927 18:47:54.858357 12289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:54.858805 12289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:54.865888 12289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7f3e4dfa6cb48b396d68813b755cae2: Sending DeleteTablet for 3 replicas of tablet 7b15ce258ff742a0802b6d3ed33f8dfe
I20260927 18:47:54.867831 12423 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b15ce258ff742a0802b6d3ed33f8dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:54 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:45728
I20260927 18:47:54.870822 12702 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b15ce258ff742a0802b6d3ed33f8dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:54 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:55112
I20260927 18:47:54.871801 12568 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b15ce258ff742a0802b6d3ed33f8dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:54 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:57626
I20260927 18:47:54.876127 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12327
I20260927 18:47:54.893697 12821 tablet_replica.cc:330] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: stopping tablet replica
I20260927 18:47:54.894210 12821 raft_consensus.cc:2229] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:54.894639 12819 tablet_replica.cc:330] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: stopping tablet replica
I20260927 18:47:54.895082 12821 raft_consensus.cc:2258] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:54.895851 12819 raft_consensus.cc:2229] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:54.897902 12821 ts_tablet_manager.cc:1905] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:54.898258 12819 raft_consensus.cc:2258] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:54.900504 12820 tablet_replica.cc:330] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: stopping tablet replica
I20260927 18:47:54.900971 12820 raft_consensus.cc:2229] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:54.901515 12820 raft_consensus.cc:2258] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:54.902599 12819 ts_tablet_manager.cc:1905] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:54.905030 12821 ts_tablet_manager.cc:1918] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:54.908560 12820 ts_tablet_manager.cc:1905] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:54.917251 12821 log.cc:1196] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/7b15ce258ff742a0802b6d3ed33f8dfe
I20260927 18:47:54.918031 12821 ts_tablet_manager.cc:1939] T 7b15ce258ff742a0802b6d3ed33f8dfe P 4160af8206304004a6a5268b7c0b13d7: Deleting consensus metadata
I20260927 18:47:54.920089 12275 catalog_manager.cc:4841] TS 4160af8206304004a6a5268b7c0b13d7 (127.31.26.194:35857): tablet 7b15ce258ff742a0802b6d3ed33f8dfe (table test-table [id=39ad9d59ece84c708646bd2c2676c07c]) successfully deleted
I20260927 18:47:54.919207 12819 ts_tablet_manager.cc:1918] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:54.920351 12819 log.cc:1196] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/7b15ce258ff742a0802b6d3ed33f8dfe
I20260927 18:47:54.920943 12819 ts_tablet_manager.cc:1939] T 7b15ce258ff742a0802b6d3ed33f8dfe P 61015ce760dc4e059b376b485aeca9b2: Deleting consensus metadata
I20260927 18:47:54.922984 12820 ts_tablet_manager.cc:1918] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:54.922981 12275 catalog_manager.cc:4841] TS 61015ce760dc4e059b376b485aeca9b2 (127.31.26.193:36841): tablet 7b15ce258ff742a0802b6d3ed33f8dfe (table test-table [id=39ad9d59ece84c708646bd2c2676c07c]) successfully deleted
I20260927 18:47:54.923180 12820 log.cc:1196] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/7b15ce258ff742a0802b6d3ed33f8dfe
I20260927 18:47:54.923696 12820 ts_tablet_manager.cc:1939] T 7b15ce258ff742a0802b6d3ed33f8dfe P e9558d2735354500836fa7d335ec20b0: Deleting consensus metadata
I20260927 18:47:54.925470 12276 catalog_manager.cc:4841] TS e9558d2735354500836fa7d335ec20b0 (127.31.26.195:37193): tablet 7b15ce258ff742a0802b6d3ed33f8dfe (table test-table [id=39ad9d59ece84c708646bd2c2676c07c]) successfully deleted
I20260927 18:47:54.926735 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:54.877561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.31.26.193:36841)
0927 18:47:54.877771 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:54.877778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:54.899009 (+ 21231us) server_negotiation.cc:407] Connection header received
0927 18:47:54.900040 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:54.900048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:54.900164 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:54.900272 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:54.903769 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.904797 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.907003 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:54.907681 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:54.907823 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:54.912656 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:54.912674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:54.912695 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:54.912743 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:54.917258 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.918520 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.918532 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.918537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.918724 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:54.919339 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:54.919346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:54.919349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:54.919874 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:54.920213 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:54.925991 (+ 5778us) server_negotiation.cc:299] Negotiation successful
0927 18:47:54.926281 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:47:54.927501 12483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:55.023214 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12327
I20260927 18:47:55.046435 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12492
I20260927 18:47:55.064998 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:55.047396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.31.26.194:35857)
0927 18:47:55.047580 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:55.047585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:55.047695 (+ 110us) server_negotiation.cc:407] Connection header received
0927 18:47:55.047898 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:55.047904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:55.047983 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:55.048085 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:55.050002 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:55.051276 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:55.053368 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:55.054108 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:55.054249 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:55.057824 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:55.057846 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:55.057876 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:55.057927 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:55.061728 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:55.062743 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:55.062751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:55.062753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:55.062932 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:55.063384 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:55.063388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:55.063390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:55.063765 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:55.064018 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:55.064418 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 18:47:55.064610 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:47:55.065873 12628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:55.177161 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12492
I20260927 18:47:55.198606 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12637
I20260927 18:47:55.213783 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:55.199441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.31.26.195:37193)
0927 18:47:55.199672 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:55.199678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:55.199700 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:47:55.199918 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:55.199924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:55.200010 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:55.200124 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:55.201715 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:55.202475 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:55.203910 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:55.204333 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:55.204479 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:55.207489 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:55.207506 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:55.207523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:55.207563 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:55.210564 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:55.211415 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:55.211423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:55.211426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:55.211581 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:55.212028 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:55.212036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:55.212044 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:55.212450 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:55.212768 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:55.213133 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 18:47:55.213368 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 18:47:55.214636 12762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:55.304416 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12637
I20260927 18:47:55.321043 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12257
I20260927 18:47:55.322382 12320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:47:55.436417 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12257
I20260927 18:47:55.458333 31851 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:47:55.794636 31851 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:47:55.794857 31851 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:47:56.141229 31851 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:47:56.141 UTC [11309] LOG: received smart shutdown request
2026-09-27 18:47:56.145 UTC [11309] LOG: background worker "logical replication launcher" (PID 11318) exited with exit code 1
2026-09-27 18:47:56.145 UTC [11313] LOG: shutting down
2026-09-27 18:47:56.340 UTC [11309] LOG: database system is shut down
2026-09-27T18:47:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55839 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:56 dist-test-slave-rfd5 krb5kdc[12835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:56 dist-test-slave-rfd5 krb5kdc[12835](info): set up 2 sockets
Sep 27 18:47:56 dist-test-slave-rfd5 krb5kdc[12835](info): commencing operation
krb5kdc: starting...
W20260927 18:47:57.574134 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:47:57 dist-test-slave-rfd5 krb5kdc[12835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:47:57.816972 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35573
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35573,127.31.26.253:35145,127.31.26.252:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:58.172907 12848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:58.173470 12848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:58.173698 12848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:58.181535 12848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:58.181664 12848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:58.181743 12848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:58.181854 12848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:58.181916 12848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:58.192502 12848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35573,127.31.26.253:35145,127.31.26.252:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:58.194288 12848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:58.195819 12848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:58.209355 12853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:58.209411 12856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:58.209338 12854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:58.210462 12848 server_base.cc:1034] running on GCE node
I20260927 18:47:58.211313 12848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:58.212833 12848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:58.214105 12848 hybrid_clock.cc:648] HybridClock initialized: now 1790534878214033 us; error 60 us; skew 500 ppm
I20260927 18:47:58.214610 12848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:58 dist-test-slave-rfd5 krb5kdc[12835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:58.223968 12848 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:47:58.225917 12848 webserver.cc:460] Webserver started at http://127.31.26.254:38373/ using document root <none> and password file <none>
I20260927 18:47:58.226712 12848 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:58.226881 12848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:58.227210 12848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:58.230597 12848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "2fd0559a4b1b4f988c823d8a749ed1e2"
format_stamp: "Formatted at 2026-09-27 18:47:58 on dist-test-slave-rfd5"
server_key: "6ca7f972959d7e750c3c921a579dd2fc"
server_key_iv: "572feaf6c6faf1279716856ae29b3f56"
server_key_version: "encryptionkey@0"
I20260927 18:47:58.231513 12848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "2fd0559a4b1b4f988c823d8a749ed1e2"
format_stamp: "Formatted at 2026-09-27 18:47:58 on dist-test-slave-rfd5"
server_key: "6ca7f972959d7e750c3c921a579dd2fc"
server_key_iv: "572feaf6c6faf1279716856ae29b3f56"
server_key_version: "encryptionkey@0"
I20260927 18:47:58.237591 12848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:47:58.241598 12863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:58.243232 12848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:58.243471 12848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "2fd0559a4b1b4f988c823d8a749ed1e2"
format_stamp: "Formatted at 2026-09-27 18:47:58 on dist-test-slave-rfd5"
server_key: "6ca7f972959d7e750c3c921a579dd2fc"
server_key_iv: "572feaf6c6faf1279716856ae29b3f56"
server_key_version: "encryptionkey@0"
I20260927 18:47:58.243717 12848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:58.260694 12848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:58.265837 12848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:58.266184 12848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:58.286566 12848 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:35573
I20260927 18:47:58.286598 12914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:35573 every 8 connection(s)
I20260927 18:47:58.288107 12848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:58.291320 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12848
I20260927 18:47:58.291574 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:47:58.292443 31851 external_mini_cluster.cc:1428] Setting key 468dd358bfb7545f2616b8307db7f8d6
I20260927 18:47:58.292986 12915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:58.303370 12915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.26.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.253@KRBTEST.COM" created.
Sep 27 18:47:58 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:58.329638 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.306757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35573 (local address 127.0.0.1:55074)
0927 18:47:58.307693 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.307744 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:58.310203 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:58.311329 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:58.311335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:58.311933 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:58.312288 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:58.312307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.314896 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.314900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:58.316229 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:58.316242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.317361 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.317367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:58.317545 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:58.318470 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:58.318508 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:58.321491 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:58.324997 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:58.325010 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:58.325024 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:58.325534 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:58.325974 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:58.325981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:58.325985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:58.326147 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:58.326965 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:58.326974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:58.327193 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 18:47:58.327447 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 18:47:58.328140 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":184,"thread_start_us":408,"threads_started":1}
I20260927 18:47:58.329658 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.308227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.31.26.254:35573)
0927 18:47:58.309000 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:58.309011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:58.309110 (+ 99us) server_negotiation.cc:407] Connection header received
0927 18:47:58.310652 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:58.310663 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:58.311047 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:58.311314 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:58.312518 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.314721 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.316406 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.317043 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.317300 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:58.321771 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:58.321780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:58.321793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:58.321827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:58.324807 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:58.325683 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:58.325692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:58.325696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:58.325853 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:58.326263 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:58.326268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:58.326271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:58.326817 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:58.327049 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:58.327582 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 18:47:58.328025 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":536,"thread_start_us":365,"threads_started":1}
I20260927 18:47:58.332927 12915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:47:58.334225 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.333754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40120)
0927 18:47:58.333960 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.334100 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 18:47:58.334532 12865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
I20260927 18:47:58.336493 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.336055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40122)
0927 18:47:58.336248 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.336334 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 18:47:58.336717 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:35145: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab.
Entry for principal kudu/127.31.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab.
I20260927 18:47:58.349737 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:35145
--webserver_interface=127.31.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab
--principal=kudu/127.31.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35573,127.31.26.253:35145,127.31.26.252:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:58.386086 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } attempt: 1
I20260927 18:47:58.387693 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.386976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40126)
0927 18:47:58.387283 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.387491 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 18:47:58.388855 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.388430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40128)
0927 18:47:58.388596 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.388686 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:47:58.389106 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:35145: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
I20260927 18:47:58.457367 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } attempt: 2
I20260927 18:47:58.458714 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.458120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40134)
0927 18:47:58.458405 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.458525 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:47:58.459933 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.459480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40144)
0927 18:47:58.459668 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.459758 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 18:47:58.460166 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:35145: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
I20260927 18:47:58.551505 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } attempt: 3
I20260927 18:47:58.552973 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.552243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40146)
0927 18:47:58.552552 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.552726 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 18:47:58.554554 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.553932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40150)
0927 18:47:58.554170 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.554300 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 18:47:58.554889 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:35145: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
I20260927 18:47:58.698227 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } attempt: 4
I20260927 18:47:58.699694 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.699003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40166)
0927 18:47:58.699386 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.699505 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260927 18:47:58.701197 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.700552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40168)
0927 18:47:58.700732 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.700981 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 18:47:58.701473 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:35145: Network error: Client connection negotiation failed: client connection to 127.31.26.253:35145: connect: Connection refused (error 111)
W20260927 18:47:58.721442 12922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:58.721750 12922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:58.721946 12922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:58.729568 12922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:58.729700 12922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:58.729794 12922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:58.729887 12922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:58.729983 12922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:58.740517 12922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35573,127.31.26.253:35145,127.31.26.252:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab
--principal=kudu/127.31.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:58.742077 12922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:58.743561 12922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:58.756953 12927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:58.756922 12930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:58.756886 12928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:58.757947 12922 server_base.cc:1034] running on GCE node
I20260927 18:47:58.758857 12922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:58.760550 12922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:58.761842 12922 hybrid_clock.cc:648] HybridClock initialized: now 1790534878761799 us; error 44 us; skew 500 ppm
I20260927 18:47:58.762285 12922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:58 dist-test-slave-rfd5 krb5kdc[12835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:58.765928 12922 init.cc:377] Logged in from keytab as kudu/127.31.26.253@KRBTEST.COM (short username kudu)
I20260927 18:47:58.767693 12922 webserver.cc:460] Webserver started at http://127.31.26.253:34645/ using document root <none> and password file <none>
I20260927 18:47:58.768466 12922 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:58.768640 12922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:58.768936 12922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:58.772131 12922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data/instance:
uuid: "a51387b7c3004fa0928a8febc414aabe"
format_stamp: "Formatted at 2026-09-27 18:47:58 on dist-test-slave-rfd5"
server_key: "a8bb64e26e130043113c1821c0161d53"
server_key_iv: "616ad4c8394c16bc4e98217c975f6f52"
server_key_version: "encryptionkey@0"
I20260927 18:47:58.773016 12922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal/instance:
uuid: "a51387b7c3004fa0928a8febc414aabe"
format_stamp: "Formatted at 2026-09-27 18:47:58 on dist-test-slave-rfd5"
server_key: "a8bb64e26e130043113c1821c0161d53"
server_key_iv: "616ad4c8394c16bc4e98217c975f6f52"
server_key_version: "encryptionkey@0"
I20260927 18:47:58.778951 12922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:47:58.783574 12937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:58.785537 12922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 18:47:58.785766 12922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal
uuid: "a51387b7c3004fa0928a8febc414aabe"
format_stamp: "Formatted at 2026-09-27 18:47:58 on dist-test-slave-rfd5"
server_key: "a8bb64e26e130043113c1821c0161d53"
server_key_iv: "616ad4c8394c16bc4e98217c975f6f52"
server_key_version: "encryptionkey@0"
I20260927 18:47:58.786018 12922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:58.827396 12922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:58.833256 12922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:58.833537 12922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:58.854893 12922 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.253:35145
I20260927 18:47:58.855006 12988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.253:35145 every 8 connection(s)
I20260927 18:47:58.856451 12922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
I20260927 18:47:58.860673 12989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:58.865588 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12922
I20260927 18:47:58.865804 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-1/wal/instance
I20260927 18:47:58.866415 31851 external_mini_cluster.cc:1428] Setting key 82914ec844392a693b16320bea3c3779
I20260927 18:47:58.869187 12989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:58 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.252@KRBTEST.COM" created.
I20260927 18:47:58.894748 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.873233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.31.26.254:35573)
0927 18:47:58.873790 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:58.873798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:58.875061 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 18:47:58.877234 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:58.877240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:58.877345 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:58.877487 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:58.879278 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.880016 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.881898 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.882326 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.882488 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:58.886956 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:58.886967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:58.886974 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:58.887042 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:58.890451 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:58.891696 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:58.891703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:58.891713 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:58.891952 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:58.892485 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:58.892491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:58.892495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:58.892860 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:58.893044 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:58.893981 (+ 937us) server_negotiation.cc:299] Negotiation successful
0927 18:47:58.894285 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
I20260927 18:47:58.895826 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.873885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35573 (local address 127.0.0.1:55080)
0927 18:47:58.874809 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.874862 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:58.876851 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:58.877772 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:58.877782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:58.878393 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:58.879021 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:58.879043 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.880226 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.880231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:58.881695 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:58.881711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.882608 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.882617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:58.882860 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:58.883741 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:58.883769 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:58.886620 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:58.890679 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:58.890688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:58.890730 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:58.891511 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:58.892090 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:58.892096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:58.892105 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:58.892326 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:58.893035 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:58.893045 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:58.893528 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 18:47:58.893866 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 18:47:58.894504 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":450,"threads_started":1}
I20260927 18:47:58.898053 12989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:58 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:47:58.917177 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.899037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40180)
0927 18:47:58.899262 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.899287 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:58.899421 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:58.900745 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:58.900754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:58.901444 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:58.901692 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:58.901711 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.903240 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.903250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:58.905201 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:58.905228 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.906276 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.906287 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:58.906582 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:58.907342 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:58.907375 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:58.910526 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:58.914278 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:58.914283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:58.914288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:58.914715 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:58.915189 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:58.915194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:58.915198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:58.915367 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:58.916089 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:58.916094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:58.916245 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 18:47:58.916404 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 18:47:58.916722 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:47:58.917320 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.899534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.31.26.253:35145)
0927 18:47:58.900226 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:58.900233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:58.900259 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:47:58.900305 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:58.900311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:58.900544 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:58.900672 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:58.901912 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.903057 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.905423 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:58.906056 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:58.906181 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:58.910786 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:58.910802 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:58.910819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:58.910866 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:58.914087 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:58.914892 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:58.914900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:58.914902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:58.915063 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:58.915493 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:58.915498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:58.915506 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:58.915948 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:58.916217 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:58.916571 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 18:47:58.916851 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":541,"thread_start_us":409,"threads_started":1}
I20260927 18:47:58.920238 12989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:47:58.921598 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.921057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51582)
0927 18:47:58.921287 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.921426 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 18:47:58.921885 12938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:58.923674 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.923215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51594)
0927 18:47:58.923452 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.923546 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260927 18:47:58.924086 12989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab.
Entry for principal kudu/127.31.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab.
I20260927 18:47:58.925522 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:42953
--webserver_interface=127.31.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab
--principal=kudu/127.31.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35573,127.31.26.253:35145,127.31.26.252:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:58.973495 12989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 1
I20260927 18:47:58.974915 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.974378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51596)
0927 18:47:58.974609 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.974723 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 18:47:58.976173 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:58.975764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51598)
0927 18:47:58.975965 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:58.976048 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 18:47:58.976600 12989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.000788 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } attempt: 5
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:47:59.016774 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.001631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40192)
0927 18:47:59.001825 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.001848 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:59.001980 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:59.002382 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:59.002385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:59.002783 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:59.002967 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.002977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.004695 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.004704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.006252 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.006274 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.007136 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.007142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.007353 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.008034 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:59.008058 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:59.010624 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:59.014030 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.014036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.014040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.014475 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.014887 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.014893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.014897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.015049 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.015606 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:59.015611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:59.015790 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 18:47:59.015953 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:47:59.016344 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 18:47:59.017958 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.001748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.31.26.253:35145)
0927 18:47:59.001981 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.001987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.002012 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:47:59.002079 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.002085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.002184 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.002309 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.003190 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.004472 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.006452 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.006935 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.007058 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.010908 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.010914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.010917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.010956 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.013840 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.014618 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.014625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.014629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.014761 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.015171 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.015176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.015179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.015478 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.015665 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.017434 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.017683 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 18:47:59.019601 12915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:47:59.021152 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.020598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51600)
0927 18:47:59.020853 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.020946 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 18:47:59.022441 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.022016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51608)
0927 18:47:59.022215 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.022311 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260927 18:47:59.022836 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.044925 12989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 2
I20260927 18:47:59.046418 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.045828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51624)
0927 18:47:59.046065 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.046205 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 18:47:59.047811 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.047348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51634)
0927 18:47:59.047561 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.047660 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:47:59.048128 12989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.074131 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 1
I20260927 18:47:59.075459 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.074959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51636)
0927 18:47:59.075171 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.075266 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:47:59.076745 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.076391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51646)
0927 18:47:59.076543 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.076625 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:47:59.077426 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.139410 12989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 3
I20260927 18:47:59.140805 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.140263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51652)
0927 18:47:59.140544 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.140641 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 18:47:59.142010 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.141563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51668)
0927 18:47:59.141806 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.141897 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260927 18:47:59.142251 12989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.145671 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 2
I20260927 18:47:59.146840 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.146366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51674)
0927 18:47:59.146589 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.146684 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 18:47:59.147966 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.147542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51682)
0927 18:47:59.147706 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.147799 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:47:59.148193 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.254491 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 3
I20260927 18:47:59.255882 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.255323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51698)
0927 18:47:59.255568 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.255683 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 18:47:59.257403 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.256855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51700)
0927 18:47:59.257177 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.257273 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 18:47:59.257728 12915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
I20260927 18:47:59.285477 12989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 4
I20260927 18:47:59.286816 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.286255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51706)
0927 18:47:59.286495 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.286616 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:47:59.288245 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.287769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51716)
0927 18:47:59.287980 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.288078 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 18:47:59.288527 12989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:42953: Network error: Client connection negotiation failed: client connection to 127.31.26.252:42953: connect: Connection refused (error 111)
W20260927 18:47:59.290900 12997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:59.291296 12997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:59.291476 12997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:59.299753 12997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:59.299950 12997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:59.300065 12997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:59.300176 12997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:59.300256 12997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:59.311087 12997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35573,127.31.26.253:35145,127.31.26.252:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab
--principal=kudu/127.31.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:47:59.312582 12997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:59.314158 12997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:59.327319 13002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:59.327479 13005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:59.327925 13003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:59.328343 12997 server_base.cc:1034] running on GCE node
I20260927 18:47:59.329330 12997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:59.330929 12997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:59.332244 12997 hybrid_clock.cc:648] HybridClock initialized: now 1790534879332183 us; error 54 us; skew 500 ppm
I20260927 18:47:59.332715 12997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:59.336659 12997 init.cc:377] Logged in from keytab as kudu/127.31.26.252@KRBTEST.COM (short username kudu)
I20260927 18:47:59.338459 12997 webserver.cc:460] Webserver started at http://127.31.26.252:38717/ using document root <none> and password file <none>
I20260927 18:47:59.339267 12997 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:59.339392 12997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:59.339758 12997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:59.343216 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data/instance:
uuid: "cfffeaf17c514fdd944080655f19f6e7"
format_stamp: "Formatted at 2026-09-27 18:47:59 on dist-test-slave-rfd5"
server_key: "48e977ae74cb24bcdbd181c2c8c080c5"
server_key_iv: "af05d907173fb6f74469bb7f8cd153bb"
server_key_version: "encryptionkey@0"
I20260927 18:47:59.344146 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal/instance:
uuid: "cfffeaf17c514fdd944080655f19f6e7"
format_stamp: "Formatted at 2026-09-27 18:47:59 on dist-test-slave-rfd5"
server_key: "48e977ae74cb24bcdbd181c2c8c080c5"
server_key_iv: "af05d907173fb6f74469bb7f8cd153bb"
server_key_version: "encryptionkey@0"
I20260927 18:47:59.350534 12997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:47:59.354821 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:59.356541 12997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:47:59.356775 12997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal
uuid: "cfffeaf17c514fdd944080655f19f6e7"
format_stamp: "Formatted at 2026-09-27 18:47:59 on dist-test-slave-rfd5"
server_key: "48e977ae74cb24bcdbd181c2c8c080c5"
server_key_iv: "af05d907173fb6f74469bb7f8cd153bb"
server_key_version: "encryptionkey@0"
I20260927 18:47:59.357038 12997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:59.378813 12997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:59.383769 12997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:59.384096 12997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:59.404124 12997 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.252:42953
I20260927 18:47:59.404161 13063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.252:42953 every 8 connection(s)
I20260927 18:47:59.405784 12997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
I20260927 18:47:59.409801 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 12997
I20260927 18:47:59.410039 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534668899567-31851-0/minicluster-data/master-2/wal/instance
I20260927 18:47:59.410101 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:59.410612 31851 external_mini_cluster.cc:1428] Setting key 62c35d845ee10e96f1fbabe8e2eaaaef
I20260927 18:47:59.420801 13064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:59.434989 12915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 4
I20260927 18:47:59.438560 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.412624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.31.26.252:42953)
0927 18:47:59.413610 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.413626 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.413683 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:47:59.415504 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.415537 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.416128 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.416651 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.418197 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.420162 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.421859 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.422857 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.423048 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.428285 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.428299 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.428321 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.428384 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.432938 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.433898 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.433907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.433915 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.434154 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.434655 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.434661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.434663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.435421 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.435741 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.436309 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.436831 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":415,"threads_started":1}
I20260927 18:47:59.443049 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.423960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35573 (local address 127.0.0.1:55084)
0927 18:47:59.424914 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.424959 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:59.425202 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:59.425642 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:59.425651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:59.426275 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:59.426563 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.426571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.428153 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.428158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.429864 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.429890 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.430947 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.430956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.431174 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.431802 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:59.431834 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:59.434620 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:59.439697 (+ 5077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.439711 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.439724 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.440365 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.440833 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.440841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.440845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.440998 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.441783 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:59.441791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:59.441980 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 18:47:59.442204 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 18:47:59.442553 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":569,"threads_started":1}
I20260927 18:47:59.444591 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.423985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.31.26.254:35573)
0927 18:47:59.424597 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.424604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.425073 (+ 469us) server_negotiation.cc:407] Connection header received
0927 18:47:59.425351 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.425357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.425443 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.425568 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.426727 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.427924 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.430064 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.430686 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.430793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.434859 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.434864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.434868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.434920 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.439513 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.440527 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.440532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.440535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.440697 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.441163 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.441169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.441173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.441617 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.441772 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.443640 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.443929 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":311,"threads_started":1}
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:47:59.446328 13064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:47:59.455686 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.435669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51740)
0927 18:47:59.435877 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.435896 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:59.436053 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:59.438447 (+ 2394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:59.438454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:59.438954 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:59.439215 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.439228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.440863 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.440868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.442310 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.442325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.443293 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.443299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.443518 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.444107 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:59.444123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:59.447110 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:59.452428 (+ 5318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.452435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.452441 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.453013 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.453532 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.453540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.453544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.453721 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.454639 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:59.454645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:59.454825 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 18:47:59.454959 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 18:47:59.455288 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:47:59.456812 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.436135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.31.26.252:42953)
0927 18:47:59.438135 (+ 2000us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.438139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.438158 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:47:59.438199 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.438202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.438261 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.438396 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.439419 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.440731 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.442457 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.443086 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.443778 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.447386 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.447393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.447398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.447440 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.452272 (+ 4832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.453187 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.453192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.453195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.453385 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.453900 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.453907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.453913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.454448 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.456149 (+ 1701us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.456324 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.456510 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"spinlock_wait_cycles":32640,"thread_start_us":1676,"threads_started":1}
I20260927 18:47:59.463565 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.447185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:35145 (local address 127.0.0.1:40194)
0927 18:47:59.447614 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.447632 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:59.447748 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:59.448277 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:59.448282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:59.448865 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:59.449128 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.449145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.450561 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.450568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.452108 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.452127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.453173 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.453180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.453433 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.454078 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:59.454102 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:59.456863 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:59.460763 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.460767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.460770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.461446 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.461970 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.461976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.461980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.462142 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.462780 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:59.462783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:59.462939 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 18:47:59.463084 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 18:47:59.463311 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260927 18:47:59.463572 12915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: Bootstrap starting.
I20260927 18:47:59.463848 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.447226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.31.26.253:35145)
0927 18:47:59.447453 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.447458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.447720 (+ 262us) server_negotiation.cc:407] Connection header received
0927 18:47:59.447886 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.447891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.447981 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.448084 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.449322 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.450396 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.452294 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.452790 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.452874 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.457172 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.457183 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.457187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.457224 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.460611 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.461586 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.461592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.461596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.461771 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.462289 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.462294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.462297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.462655 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.462815 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.463245 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.463475 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:47:59.464852 13064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:47:59.467736 12915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:59.469257 12915 log.cc:824] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:59.472412 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.442652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.31.26.254:35573)
0927 18:47:59.444881 (+ 2229us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.444886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.444902 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:47:59.444958 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.444962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.445113 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.445240 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.446892 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.447945 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.449705 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.450335 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.450486 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.455096 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.455102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.455106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.455137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.468495 (+ 13358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.469464 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.469469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.469474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.469662 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.470230 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.470236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.470239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.470669 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.471006 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.471777 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.472029 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2110,"thread_start_us":4202,"threads_started":1}
I20260927 18:47:59.473275 12915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: No bootstrap required, opened a new log
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:47:59.480474 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.465719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.31.26.252:42953)
0927 18:47:59.465912 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.465916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.465931 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:47:59.466096 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.466099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.466155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.466230 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.467174 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.467938 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.469641 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.470340 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.470488 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.474373 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.474380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.474384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.474424 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.477644 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.478423 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.478428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.478431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.478594 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.478938 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.478946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.478949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.479206 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.479384 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.479747 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.480130 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":53}
I20260927 18:47:59.480494 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.465649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51752)
0927 18:47:59.465839 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.465859 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:59.465975 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:59.466287 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:59.466292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:59.466784 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:59.467008 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.467022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.468079 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.468087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.469491 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.469508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.470535 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.470542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.470747 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.471386 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:59.471404 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:59.474178 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:59.477802 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.477808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.477812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.478299 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.478681 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.478687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.478691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.478826 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.479328 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:59.479337 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:59.479485 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 18:47:59.479649 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 18:47:59.480020 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":4}
I20260927 18:47:59.484359 12915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.484879 12915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:59.485002 12915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd0559a4b1b4f988c823d8a749ed1e2, State: Initialized, Role: FOLLOWER
I20260927 18:47:59.485602 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: Bootstrap starting.
I20260927 18:47:59.485854 12915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:47:59.489599 13064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:59.491122 13064 log.cc:824] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:59.493196 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.493645 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.495203 12915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:59.495385 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: No bootstrap required, opened a new log
I20260927 18:47:59.496608 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.474910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.31.26.253:35145)
0927 18:47:59.475122 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.475130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.475225 (+ 95us) server_negotiation.cc:407] Connection header received
0927 18:47:59.475431 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.475438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.475528 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.475645 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.477337 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.478470 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.480392 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.481236 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.481337 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.488137 (+ 6800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.488145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.488149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.488228 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.492263 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.493296 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.493302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.493305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.493505 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.494033 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.494038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.494040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.494484 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.495835 (+ 1351us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.495995 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.496261 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:47:59.503464 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:47:59.506640 13064 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.507136 13064 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:59.507266 13064 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfffeaf17c514fdd944080655f19f6e7, State: Initialized, Role: FOLLOWER
I20260927 18:47:59.507947 13064 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
W20260927 18:47:59.517988 13085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:59.518321 13085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:59.519164 13089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
W20260927 18:47:59.519546 12869 tablet.cc:2369] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:59.519791 13089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.520802 13064 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:59.508771 12848 master_runner.cc:186] Error getting master registration for 127.31.26.253:35145: OK, instance_id { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" instance_seqno: 1790534878848690 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:47:59.527220 12848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:47:59.561367 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.530016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.31.26.252:42953)
0927 18:47:59.530203 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.530208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.538226 (+ 8018us) server_negotiation.cc:407] Connection header received
0927 18:47:59.538785 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.538792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.538883 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.538958 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.541628 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.542799 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.545409 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.546141 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.546241 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.553593 (+ 7352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.553603 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.553610 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.553699 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.557484 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.558561 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.558567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.558573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.558766 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.559241 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.559245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.559248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.559648 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.559948 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.560601 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.560851 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:47:59.562575 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.522020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.31.26.254:35573)
0927 18:47:59.522284 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.522294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.529812 (+ 7518us) server_negotiation.cc:407] Connection header received
0927 18:47:59.530082 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.530090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.530183 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.530265 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.535560 (+ 5295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.537268 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.543296 (+ 6028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.544064 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.544179 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.552543 (+ 8364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.552554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.552565 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.552620 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.558326 (+ 5706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.559249 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.559254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.559258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.559436 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.560628 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.560633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.560636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.561029 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.561355 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.561925 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.562185 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:47:59.566118 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.523214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.31.26.253:35145)
0927 18:47:59.523472 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.523482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.530661 (+ 7179us) server_negotiation.cc:407] Connection header received
0927 18:47:59.531035 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.531044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.531158 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.531266 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.539817 (+ 8551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.541220 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.544709 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.545798 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.545908 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.550408 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.550419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.550426 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.550484 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.558201 (+ 7717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.563254 (+ 5053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.563261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.563264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.563508 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.564073 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.564079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.564082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.564489 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.564739 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.565293 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.565557 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 18:47:59.575487 13105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:59.575670 13105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:59.587800 12989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } attempt: 5
I20260927 18:47:59.579226 12997 master_runner.cc:186] Error getting master registration for 127.31.26.253:35145: OK, instance_id { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" instance_seqno: 1790534878848690 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:47:59.592443 12997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 18:47:59 dist-test-slave-rfd5 krb5kdc[12835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534878, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:47:59.608549 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.588483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:42953 (local address 127.0.0.1:51774)
0927 18:47:59.588917 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:59.588939 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:59.589063 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:59.589744 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:59.589749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:59.590339 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:59.590593 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.590606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.592117 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.592122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.593646 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:59.593666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.594991 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.594999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:59.595323 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.596198 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:59.596230 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:59.599416 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:59.604701 (+ 5285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.604707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.604712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.605855 (+ 1143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.606430 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:59.606436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:59.606440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:59.606642 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:59.607402 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:59.607410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:59.607591 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 18:47:59.607760 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:47:59.608122 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260927 18:47:59.609687 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:59.589242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.31.26.252:42953)
0927 18:47:59.589417 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:59.589424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:59.589442 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:47:59.589509 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:59.589513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:59.589577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:59.589705 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:59.590738 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.591949 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.593825 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:59.594763 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:59.594927 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:59.599748 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:59.599758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:59.599764 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:59.599933 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:59.604468 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.606058 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.606067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.606070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.606265 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:59.606815 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:59.606821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:59.606825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:59.607238 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:59.607406 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:59.608971 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0927 18:47:59.609294 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:47:59.616132 12989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: Bootstrap starting.
I20260927 18:47:59.620639 12989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:59.622200 13089 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:59.622627 12989 log.cc:824] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:59.622577 13089 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.624300 13089 leader_election.cc:288] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fd0559a4b1b4f988c823d8a749ed1e2 (127.31.26.254:35573), a51387b7c3004fa0928a8febc414aabe (127.31.26.253:35145)
I20260927 18:47:59.625497 12890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfffeaf17c514fdd944080655f19f6e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" is_pre_election: true
I20260927 18:47:59.626168 12890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfffeaf17c514fdd944080655f19f6e7 in term 0.
I20260927 18:47:59.626924 13013 leader_election.cc:302] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd0559a4b1b4f988c823d8a749ed1e2, cfffeaf17c514fdd944080655f19f6e7; no voters:
I20260927 18:47:59.627442 13089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:59.627602 13089 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:59.627707 13089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:59.631412 13089 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.632823 12890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfffeaf17c514fdd944080655f19f6e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2"
I20260927 18:47:59.633126 12890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:47:59.636363 13018 tablet.cc:2369] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:59.636799 12890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfffeaf17c514fdd944080655f19f6e7 in term 1.
I20260927 18:47:59.637379 13013 leader_election.cc:302] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd0559a4b1b4f988c823d8a749ed1e2, cfffeaf17c514fdd944080655f19f6e7; no voters:
I20260927 18:47:59.638866 12989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: No bootstrap required, opened a new log
I20260927 18:47:59.642112 13089 leader_election.cc:288] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [CANDIDATE]: Term 1 election: Requested vote from peers 2fd0559a4b1b4f988c823d8a749ed1e2 (127.31.26.254:35573), a51387b7c3004fa0928a8febc414aabe (127.31.26.253:35145)
I20260927 18:47:59.642369 13089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:59.642521 13089 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [term 1 LEADER]: Becoming Leader. State: Replica: cfffeaf17c514fdd944080655f19f6e7, State: Running, Role: LEADER
I20260927 18:47:59.643088 13089 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.655753 13108 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfffeaf17c514fdd944080655f19f6e7. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.656287 13108 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:59.662092 12989 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.663568 12989 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:59.663946 12989 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51387b7c3004fa0928a8febc414aabe, State: Initialized, Role: FOLLOWER
I20260927 18:47:59.666127 12989 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } }
I20260927 18:47:59.670729 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.671185 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.671845 13110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:59.674357 13110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:59.674265 12964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfffeaf17c514fdd944080655f19f6e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51387b7c3004fa0928a8febc414aabe" is_pre_election: true
I20260927 18:47:59.674917 12964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfffeaf17c514fdd944080655f19f6e7 in term 0.
I20260927 18:47:59.675340 12963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfffeaf17c514fdd944080655f19f6e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51387b7c3004fa0928a8febc414aabe"
I20260927 18:47:59.675570 12963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:59.679131 12963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfffeaf17c514fdd944080655f19f6e7 in term 1.
I20260927 18:47:59.680487 12989 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:59.694207 12963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [term 1 FOLLOWER]: Refusing update from remote peer cfffeaf17c514fdd944080655f19f6e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:59.695446 13108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:59.704290 12890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [term 1 FOLLOWER]: Refusing update from remote peer cfffeaf17c514fdd944080655f19f6e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:59.705837 13108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:59.728206 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfffeaf17c514fdd944080655f19f6e7. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.728624 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.739073 13112 mvcc.cc:205] Tried to move back new op lower bound from 7334030867211038720 to 7334030867023331328. Current Snapshot: MvccSnapshot[applied={T|T < 7334030867211038720}]
I20260927 18:47:59.747822 13108 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.748153 13108 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:59.750779 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfffeaf17c514fdd944080655f19f6e7. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.751171 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.753283 13110 catalog_manager.cc:1347] Generated new cluster ID: 1fac46a261f04767a969cce801aca43e
I20260927 18:47:59.753425 13110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:59.766196 13089 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.766589 13089 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:59.766995 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.767315 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.773118 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfffeaf17c514fdd944080655f19f6e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fd0559a4b1b4f988c823d8a749ed1e2" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35573 } } peers { permanent_uuid: "a51387b7c3004fa0928a8febc414aabe" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 35145 } } peers { permanent_uuid: "cfffeaf17c514fdd944080655f19f6e7" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 42953 } } }
I20260927 18:47:59.773422 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:59.780108 13118 mvcc.cc:205] Tried to move back new op lower bound from 7334030867211038720 to 7334030867023331328. Current Snapshot: MvccSnapshot[applied={T|T < 7334030867211038720 or (T in {7334030867211038720})}]
I20260927 18:47:59.780340 13110 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:59.782622 13110 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:59.791388 13130 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:47:59.817276 13110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7: Generated new TSK 0
I20260927 18:47:59.818471 13110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:48:00.520483 13085 catalog_manager.cc:1259] Loaded cluster ID: 1fac46a261f04767a969cce801aca43e
I20260927 18:48:00.520598 13085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: loading cluster ID for follower catalog manager: success
I20260927 18:48:00.523622 13085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: acquiring CA information for follower catalog manager: success
I20260927 18:48:00.525648 13085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2fd0559a4b1b4f988c823d8a749ed1e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:48:00.796551 13130 catalog_manager.cc:1259] Loaded cluster ID: 1fac46a261f04767a969cce801aca43e
I20260927 18:48:00.796657 13130 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: loading cluster ID for follower catalog manager: success
I20260927 18:48:00.799988 13130 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: acquiring CA information for follower catalog manager: success
I20260927 18:48:00.802367 13130 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a51387b7c3004fa0928a8febc414aabe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:48:00.922180 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12848
I20260927 18:48:00.923559 12910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:01.036474 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12848
I20260927 18:48:01.058754 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12922
I20260927 18:48:01.060106 12984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:01.109766 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:01.108819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35573 (local address 127.0.0.1:50796)
0927 18:48:01.109452 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:01.109589 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":404,"threads_started":1}
W20260927 18:48:01.110070 13013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.254:35573: connect: Connection refused (error 111)
I20260927 18:48:01.111820 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:01.111429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35573 (local address 127.0.0.1:50800)
0927 18:48:01.111596 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:01.111696 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:48:01.112053 13013 consensus_peers.cc:487] T 00000000000000000000000000000000 P cfffeaf17c514fdd944080655f19f6e7 -> Peer 2fd0559a4b1b4f988c823d8a749ed1e2 (127.31.26.254:35573): Couldn't send request to peer 2fd0559a4b1b4f988c823d8a749ed1e2. Status: Network error: Client connection negotiation failed: client connection to 127.31.26.254:35573: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:48:01.176968 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12922
I20260927 18:48:01.196761 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 12997
I20260927 18:48:01.197961 13059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:01.308727 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 12997
2026-09-27T18:48:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4842 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:48:01 dist-test-slave-rfd5 krb5kdc[13142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:48:01 dist-test-slave-rfd5 krb5kdc[13142](info): set up 2 sockets
Sep 27 18:48:01 dist-test-slave-rfd5 krb5kdc[13142](info): commencing operation
krb5kdc: starting...
W20260927 18:48:02.425760 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:48:02 dist-test-slave-rfd5 krb5kdc[13142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:48:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:48:02.677433 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:45101
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:03.029848 13155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:03.030157 13155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:03.030340 13155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:03.038136 13155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:03.038282 13155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:03.038371 13155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:03.038466 13155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:03.038508 13155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:03.048928 13155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:03.050414 13155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:03.051879 13155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:03.064950 13163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:03.064922 13160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:03.064899 13161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:03.066249 13155 server_base.cc:1034] running on GCE node
I20260927 18:48:03.067324 13155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:03.068998 13155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:03.070307 13155 hybrid_clock.cc:648] HybridClock initialized: now 1790534883070248 us; error 59 us; skew 500 ppm
I20260927 18:48:03.070729 13155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:03 dist-test-slave-rfd5 krb5kdc[13142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534883, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:03.074379 13155 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:48:03.076184 13155 webserver.cc:460] Webserver started at http://127.31.26.254:39431/ using document root <none> and password file <none>
I20260927 18:48:03.077003 13155 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:03.077214 13155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:03.077569 13155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:03.080862 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "d01bdd2c37014627ab5aaf20dec387fa"
format_stamp: "Formatted at 2026-09-27 18:48:03 on dist-test-slave-rfd5"
server_key: "7430f6b5bbc13388e3ea10fad1d70996"
server_key_iv: "17998cd9cc632eed825f8eddadff110c"
server_key_version: "encryptionkey@0"
I20260927 18:48:03.081835 13155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "d01bdd2c37014627ab5aaf20dec387fa"
format_stamp: "Formatted at 2026-09-27 18:48:03 on dist-test-slave-rfd5"
server_key: "7430f6b5bbc13388e3ea10fad1d70996"
server_key_iv: "17998cd9cc632eed825f8eddadff110c"
server_key_version: "encryptionkey@0"
I20260927 18:48:03.087739 13155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:48:03.092010 13170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:03.093803 13155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:03.094061 13155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "d01bdd2c37014627ab5aaf20dec387fa"
format_stamp: "Formatted at 2026-09-27 18:48:03 on dist-test-slave-rfd5"
server_key: "7430f6b5bbc13388e3ea10fad1d70996"
server_key_iv: "17998cd9cc632eed825f8eddadff110c"
server_key_version: "encryptionkey@0"
I20260927 18:48:03.094326 13155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:03.137470 13155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:03.142406 13155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:03.142771 13155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:03.163379 13155 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:45101
I20260927 18:48:03.163403 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:45101 every 8 connection(s)
I20260927 18:48:03.165042 13155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:03.169472 13222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:03.172909 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13155
I20260927 18:48:03.173187 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:48:03.174180 31851 external_mini_cluster.cc:1428] Setting key 5e1adc9f91eb19a2c9c03ad0fbfd23bc
I20260927 18:48:03.184088 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa: Bootstrap starting.
I20260927 18:48:03.187834 13222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:03.189005 13222 log.cc:824] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:48:03 dist-test-slave-rfd5 krb5kdc[13142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:03.192638 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa: No bootstrap required, opened a new log
I20260927 18:48:03.200670 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:03.176619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.31.26.254:45101)
0927 18:48:03.177679 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:03.177694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:03.177741 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:48:03.179583 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:03.179609 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:03.180059 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:03.180548 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:03.182240 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:03.183849 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:03.185570 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:03.186372 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:03.186643 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:03.191760 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:03.191769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:03.191786 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:03.191825 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:03.195443 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:03.196553 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:03.196578 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:03.196585 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:03.196787 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:03.197293 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:03.197297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:03.197299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:03.198063 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:03.198386 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:03.198934 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 18:48:03.199466 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":509,"threads_started":1}
I20260927 18:48:03.202593 13222 raft_consensus.cc:348] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01bdd2c37014627ab5aaf20dec387fa" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45101 } }
I20260927 18:48:03.203065 13222 raft_consensus.cc:374] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:03.203239 13222 raft_consensus.cc:729] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01bdd2c37014627ab5aaf20dec387fa, State: Initialized, Role: FOLLOWER
I20260927 18:48:03.204067 13222 consensus_queue.cc:260] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01bdd2c37014627ab5aaf20dec387fa" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45101 } }
I20260927 18:48:03.204411 13222 raft_consensus.cc:388] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:03.204579 13222 raft_consensus.cc:482] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:03.204826 13222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:03.205588 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:48:03.208422 13222 raft_consensus.cc:504] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01bdd2c37014627ab5aaf20dec387fa" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45101 } }
I20260927 18:48:03.209259 13222 leader_election.cc:302] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d01bdd2c37014627ab5aaf20dec387fa; no voters:
I20260927 18:48:03.210057 13222 leader_election.cc:288] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:48:03.210327 13227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:03.211063 13227 raft_consensus.cc:686] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [term 1 LEADER]: Becoming Leader. State: Replica: d01bdd2c37014627ab5aaf20dec387fa, State: Running, Role: LEADER
I20260927 18:48:03.211644 13227 consensus_queue.cc:237] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01bdd2c37014627ab5aaf20dec387fa" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45101 } }
I20260927 18:48:03.212426 13222 sys_catalog.cc:564] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:03.214874 13228 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d01bdd2c37014627ab5aaf20dec387fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01bdd2c37014627ab5aaf20dec387fa" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45101 } } }
I20260927 18:48:03.214956 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d01bdd2c37014627ab5aaf20dec387fa. Latest consensus state: current_term: 1 leader_uuid: "d01bdd2c37014627ab5aaf20dec387fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01bdd2c37014627ab5aaf20dec387fa" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45101 } } }
I20260927 18:48:03.215516 13228 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:03.215515 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:03.216729 13236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:03.223556 13236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:03.234167 13236 catalog_manager.cc:1347] Generated new cluster ID: 455bf188da1140caaec0813f7fc0ff29
I20260927 18:48:03.234339 13236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:48:03.245644 13236 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:48:03.247102 13236 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:48:03.255188 13236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d01bdd2c37014627ab5aaf20dec387fa: Generated new TSK 0
I20260927 18:48:03.256227 13236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:48:03.268429 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13155
I20260927 18:48:03.269857 13217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:03.363593 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13155
2026-09-27T18:48:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2049 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:48:03 dist-test-slave-rfd5 krb5kdc[13250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:48:03 dist-test-slave-rfd5 krb5kdc[13250](info): set up 2 sockets
Sep 27 18:48:03 dist-test-slave-rfd5 krb5kdc[13250](info): commencing operation
krb5kdc: starting...
W20260927 18:48:04.477838 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:48:04 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:48:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:48:04.733059 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44371
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:05.089607 13263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:05.089933 13263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:05.090119 13263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:05.097707 13263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:05.097831 13263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:05.097947 13263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:05.098061 13263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:05.098151 13263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:05.108791 13263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:05.110364 13263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:05.111783 13263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:05.125061 13268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:05.125044 13269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:05.125217 13271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:05.126027 13263 server_base.cc:1034] running on GCE node
I20260927 18:48:05.126969 13263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:05.128629 13263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:05.129850 13263 hybrid_clock.cc:648] HybridClock initialized: now 1790534885129795 us; error 42 us; skew 500 ppm
I20260927 18:48:05.130249 13263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:05 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:05.133725 13263 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:48:05.135555 13263 webserver.cc:460] Webserver started at http://127.31.26.254:46241/ using document root <none> and password file <none>
I20260927 18:48:05.136332 13263 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:05.136487 13263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:05.136834 13263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:05.140075 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "d50af23098b24f6faffef23d08d1fde7"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "66ad46ffb2a24f6208267dcb6ed4e603"
server_key_iv: "b6e65a5ed748ced0676961439fe6ae2f"
server_key_version: "encryptionkey@0"
I20260927 18:48:05.140910 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "d50af23098b24f6faffef23d08d1fde7"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "66ad46ffb2a24f6208267dcb6ed4e603"
server_key_iv: "b6e65a5ed748ced0676961439fe6ae2f"
server_key_version: "encryptionkey@0"
I20260927 18:48:05.146610 13263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:48:05.150488 13278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:05.152070 13263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:48:05.152300 13263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "d50af23098b24f6faffef23d08d1fde7"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "66ad46ffb2a24f6208267dcb6ed4e603"
server_key_iv: "b6e65a5ed748ced0676961439fe6ae2f"
server_key_version: "encryptionkey@0"
I20260927 18:48:05.152539 13263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:05.177819 13263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:05.182283 13263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:05.182615 13263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:05.202630 13263 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44371
I20260927 18:48:05.202683 13329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44371 every 8 connection(s)
I20260927 18:48:05.204160 13263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:05.208259 13330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:05.209154 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13263
I20260927 18:48:05.209378 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:48:05.210263 31851 external_mini_cluster.cc:1428] Setting key 4c876cd598886548220c57e144fecc29
I20260927 18:48:05.224368 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap starting.
Sep 27 18:48:05 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:05.228298 13330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:05.229954 13330 log.cc:824] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:05.233868 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: No bootstrap required, opened a new log
I20260927 18:48:05.238055 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:05.212680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.31.26.254:44371)
0927 18:48:05.213624 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:05.213637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:05.213686 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:48:05.215416 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:05.215443 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:05.215915 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:05.216285 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:05.218194 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:05.219532 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:05.221822 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:05.223009 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:05.223221 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:05.228093 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:05.228110 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:05.228134 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:05.228205 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:05.232370 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:05.233331 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:05.233341 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:05.233349 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:05.233603 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:05.234193 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:05.234198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:05.234201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:05.235044 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:05.235398 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:05.235878 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 18:48:05.236461 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":464,"threads_started":1}
I20260927 18:48:05.244200 13330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:05.244642 13330 raft_consensus.cc:374] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:05.244721 13330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50af23098b24f6faffef23d08d1fde7, State: Initialized, Role: FOLLOWER
I20260927 18:48:05.245402 13330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:05.245685 13330 raft_consensus.cc:388] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:05.245878 13330 raft_consensus.cc:482] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:05.246155 13330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:05.248668 13330 raft_consensus.cc:504] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:05.249353 13330 leader_election.cc:302] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d50af23098b24f6faffef23d08d1fde7; no voters:
I20260927 18:48:05.249955 13330 leader_election.cc:288] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:48:05.250177 13335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:05.251396 13335 raft_consensus.cc:686] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 LEADER]: Becoming Leader. State: Replica: d50af23098b24f6faffef23d08d1fde7, State: Running, Role: LEADER
I20260927 18:48:05.252112 13335 consensus_queue.cc:237] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:05.252540 13330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:05.256270 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50af23098b24f6faffef23d08d1fde7. Latest consensus state: current_term: 1 leader_uuid: "d50af23098b24f6faffef23d08d1fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } } }
I20260927 18:48:05.256724 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:05.256623 13336 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d50af23098b24f6faffef23d08d1fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } } }
I20260927 18:48:05.256939 13336 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:05.257761 13342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:05.264853 13342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:05.279001 13342 catalog_manager.cc:1347] Generated new cluster ID: 197ccb8ad0dd4dbe9cd97d1c65804580
I20260927 18:48:05.279181 13342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:48:05.297909 13342 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:48:05.300030 13342 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:48:05.319912 13342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Generated new TSK 0
I20260927 18:48:05.320850 13342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:48:05.386266 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44371
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:05.776216 13356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:05.776641 13356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:05.776847 13356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:05.786115 13356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:05.786309 13356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:05.786509 13356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:48:05.798588 13356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:05.800128 13356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:05.801775 13356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:05.819267 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:05.819301 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:05.819527 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:05.820446 13356 server_base.cc:1034] running on GCE node
I20260927 18:48:05.821235 13356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:05.822448 13356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:05.823768 13356 hybrid_clock.cc:648] HybridClock initialized: now 1790534885823716 us; error 45 us; skew 500 ppm
I20260927 18:48:05.824185 13356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:05 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:05.828248 13356 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:48:05.830154 13356 webserver.cc:460] Webserver started at http://127.31.26.193:39081/ using document root <none> and password file <none>
I20260927 18:48:05.830983 13356 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:05.831154 13356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:05.831492 13356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:05.834895 13356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "6bf449957ed34859877548b9345b0c1a"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "a79cc65c831e087cea422121325e7311"
server_key_iv: "519a84e97fb012d147a88dba14c167ff"
server_key_version: "encryptionkey@0"
I20260927 18:48:05.835857 13356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "6bf449957ed34859877548b9345b0c1a"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "a79cc65c831e087cea422121325e7311"
server_key_iv: "519a84e97fb012d147a88dba14c167ff"
server_key_version: "encryptionkey@0"
I20260927 18:48:05.842423 13356 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:48:05.846786 13371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:05.848505 13356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:48:05.848839 13356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "6bf449957ed34859877548b9345b0c1a"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "a79cc65c831e087cea422121325e7311"
server_key_iv: "519a84e97fb012d147a88dba14c167ff"
server_key_version: "encryptionkey@0"
I20260927 18:48:05.849161 13356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:05.872588 13356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:05.877813 13356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:05.878151 13356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:05.879225 13356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:05.880815 13356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:05.880944 13356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:05.881134 13356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:05.881217 13356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 18:48:05.919234 13356 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34073
I20260927 18:48:05.919276 13483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34073 every 8 connection(s)
I20260927 18:48:05.920854 13356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:48:05.926465 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13356
I20260927 18:48:05.926712 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:48:05.927428 31851 external_mini_cluster.cc:1428] Setting key 8db6ec76a9342256c0680b0b1874593b
Sep 27 18:48:05 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:05.945616 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:05.923925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:42307 (local address 127.31.26.254:44371)
0927 18:48:05.924461 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:05.924466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:05.925441 (+ 975us) server_negotiation.cc:407] Connection header received
0927 18:48:05.927635 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:05.927641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:05.927749 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:05.927892 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:05.929805 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:05.930560 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:05.932299 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:05.932731 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:05.932896 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:05.937305 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:05.937316 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:05.937325 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:05.937379 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:05.941319 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:05.942383 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:05.942393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:05.942400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:05.942583 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:05.943226 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:05.943236 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:05.943242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:05.943689 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:05.943881 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:05.944787 (+ 906us) server_negotiation.cc:299] Negotiation successful
0927 18:48:05.945079 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260927 18:48:05.946475 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:05.924399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:42307)
0927 18:48:05.925256 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:05.925303 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:05.927296 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:05.928184 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:05.928194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:05.928783 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:05.929607 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:05.929631 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:05.930755 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:05.930759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:05.932102 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:05.932117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:05.933020 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:05.933028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:05.933332 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:05.934266 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:05.934298 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:05.936924 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:05.941573 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:05.941583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:05.941602 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:05.942187 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:05.942784 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:05.942788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:05.942791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:05.943054 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:05.943873 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:05.943880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:05.944283 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 18:48:05.944595 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 18:48:05.945065 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":304,"threads_started":1}
I20260927 18:48:05.948666 13484 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44371
I20260927 18:48:05.949120 13484 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:05.950268 13484 heartbeater.cc:511] Master 127.31.26.254:44371 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:48:05.953334 13295 ts_manager.cc:194] Registered new tserver with Master: 6bf449957ed34859877548b9345b0c1a (127.31.26.193:34073)
I20260927 18:48:05.955102 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:42307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:48:05.985571 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44371
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:06.349448 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:06.349766 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:06.349941 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:06.358093 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:06.358229 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:06.358412 13490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:48:06.368875 13490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:06.370710 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:06.372298 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:06.387151 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:06.387209 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:06.387246 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:06.387725 13490 server_base.cc:1034] running on GCE node
I20260927 18:48:06.388478 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:06.389822 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:06.391117 13490 hybrid_clock.cc:648] HybridClock initialized: now 1790534886391058 us; error 73 us; skew 500 ppm
I20260927 18:48:06.391521 13490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:06 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:06.395495 13490 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:48:06.397220 13490 webserver.cc:460] Webserver started at http://127.31.26.194:43235/ using document root <none> and password file <none>
I20260927 18:48:06.397949 13490 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:06.398057 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:06.398312 13490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:06.401556 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "829f46b3dfd86f421240391638734f3b"
server_key_iv: "9a978fcc636d16cee8c91f140ac9b512"
server_key_version: "encryptionkey@0"
I20260927 18:48:06.402472 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "829f46b3dfd86f421240391638734f3b"
server_key_iv: "9a978fcc636d16cee8c91f140ac9b512"
server_key_version: "encryptionkey@0"
I20260927 18:48:06.408597 13490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:48:06.413380 13505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:06.415241 13490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:48:06.415477 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "829f46b3dfd86f421240391638734f3b"
server_key_iv: "9a978fcc636d16cee8c91f140ac9b512"
server_key_version: "encryptionkey@0"
I20260927 18:48:06.415704 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:06.433940 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:06.438824 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:06.439147 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:06.440220 13490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:06.441807 13490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:06.441887 13490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:06.442026 13490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:06.442094 13490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:06.478888 13490 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:38785
I20260927 18:48:06.478953 13617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:38785 every 8 connection(s)
I20260927 18:48:06.480301 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:48:06.482851 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13490
I20260927 18:48:06.483093 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:48:06.483592 31851 external_mini_cluster.cc:1428] Setting key a8b56c99f5f24568386a133c12596511
Sep 27 18:48:06 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:06.505393 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:06.484317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:50695 (local address 127.31.26.254:44371)
0927 18:48:06.484781 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:06.484787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:06.484900 (+ 113us) server_negotiation.cc:407] Connection header received
0927 18:48:06.487004 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:06.487014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:06.487115 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:06.487249 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:06.489073 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:06.490359 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:06.492199 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:06.492934 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:06.493051 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:06.498190 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:06.498198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:06.498201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:06.498244 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:06.501411 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:06.502332 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:06.502339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:06.502342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:06.502503 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:06.503074 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:06.503084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:06.503091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:06.503543 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:06.503730 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:06.504582 (+ 852us) server_negotiation.cc:299] Negotiation successful
0927 18:48:06.504890 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":795,"threads_started":1}
I20260927 18:48:06.506160 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:06.483834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:50695)
0927 18:48:06.484724 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:06.484782 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:06.486678 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:06.487543 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:06.487553 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:06.488172 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:06.488838 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:06.488859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:06.490588 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:06.490593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:06.491992 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:06.492009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:06.493263 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:06.493272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:06.493687 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:06.494595 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:06.494638 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:06.497831 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:06.501624 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:06.501635 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:06.501656 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:06.502171 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:06.502640 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:06.502645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:06.502649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:06.502863 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:06.503702 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:06.503709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:06.504118 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 18:48:06.504382 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 18:48:06.504831 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":516,"threads_started":1}
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:48:06.507931 13618 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44371
I20260927 18:48:06.508289 13618 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:06.509294 13618 heartbeater.cc:511] Master 127.31.26.254:44371 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:48:06.511300 13295 ts_manager.cc:194] Registered new tserver with Master: d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785)
I20260927 18:48:06.512717 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:50695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:48:06.542214 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44371
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:06.899150 13624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:06.899588 13624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:06.899786 13624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:06.907733 13624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:06.907959 13624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:06.908175 13624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:48:06.919356 13624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:06.921191 13624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:06.922989 13624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:06.938581 13629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:06.938515 13630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:06.938562 13632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:06.939595 13624 server_base.cc:1034] running on GCE node
I20260927 18:48:06.940310 13624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:06.941501 13624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:06.942799 13624 hybrid_clock.cc:648] HybridClock initialized: now 1790534886942729 us; error 63 us; skew 500 ppm
I20260927 18:48:06.943200 13624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:06 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:06.946841 13624 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:48:06.948573 13624 webserver.cc:460] Webserver started at http://127.31.26.195:41841/ using document root <none> and password file <none>
I20260927 18:48:06.949452 13624 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:06.949615 13624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:06.949951 13624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:06.953393 13624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "364356cf0b6149fc8aeea252bb2dca90"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "ba6a25c7339bf4b13a5090efe383680f"
server_key_iv: "a269a9524911c4d1c0288d3f764387a2"
server_key_version: "encryptionkey@0"
I20260927 18:48:06.954456 13624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "364356cf0b6149fc8aeea252bb2dca90"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "ba6a25c7339bf4b13a5090efe383680f"
server_key_iv: "a269a9524911c4d1c0288d3f764387a2"
server_key_version: "encryptionkey@0"
I20260927 18:48:06.959267 13484 heartbeater.cc:503] Master 127.31.26.254:44371 was elected leader, sending a full tablet report...
I20260927 18:48:06.960915 13624 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:48:06.964923 13639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:06.966697 13624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:48:06.966909 13624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "364356cf0b6149fc8aeea252bb2dca90"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "ba6a25c7339bf4b13a5090efe383680f"
server_key_iv: "a269a9524911c4d1c0288d3f764387a2"
server_key_version: "encryptionkey@0"
I20260927 18:48:06.967146 13624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:06.987424 13624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:06.991963 13624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:06.992296 13624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:06.993450 13624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:06.995008 13624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:06.995131 13624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:06.995291 13624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:06.995369 13624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:07.031980 13624 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:45089
I20260927 18:48:07.032011 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:45089 every 8 connection(s)
I20260927 18:48:07.033473 13624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:48:07.038396 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13624
I20260927 18:48:07.038622 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:48:07.039186 31851 external_mini_cluster.cc:1428] Setting key 90400fed19b1de9b107abac5c9a94225
Sep 27 18:48:07 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534886, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:07.056607 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.036299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:54865 (local address 127.31.26.254:44371)
0927 18:48:07.036816 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.036820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.037865 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 18:48:07.040636 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.040641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.040795 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.040928 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:07.042684 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.043488 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.045266 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.045692 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.045844 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.049771 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:07.049778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:07.049781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:07.049819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:07.052647 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.053792 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.053804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.053806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.054034 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.054584 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.054602 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.054604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.054910 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:07.055105 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.055916 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.056165 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260927 18:48:07.057708 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.036845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:54865)
0927 18:48:07.037672 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.037721 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:07.040286 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:07.041261 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:07.041271 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:07.041843 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:07.042487 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.042507 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.043683 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.043689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.045018 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.045032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.045976 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.045984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.046247 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.046970 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:07.046998 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:07.049462 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:07.052879 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:07.052889 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:07.052910 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:07.053634 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:07.054202 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:07.054208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:07.054212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:07.054431 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:07.055065 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:07.055075 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:07.055494 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 18:48:07.055828 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 18:48:07.056449 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260927 18:48:07.059217 13752 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44371
I20260927 18:48:07.059604 13752 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:07.060454 13752 heartbeater.cc:511] Master 127.31.26.254:44371 requested a full tablet report, sending...
I20260927 18:48:07.062197 13295 ts_manager.cc:194] Registered new tserver with Master: 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089)
I20260927 18:48:07.063102 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:54865
I20260927 18:48:07.068377 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:48:07.087097 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.073520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.31.26.254:44371)
0927 18:48:07.073807 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.073814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.074157 (+ 343us) server_negotiation.cc:407] Connection header received
0927 18:48:07.074397 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.074401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.074483 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.074611 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:07.076126 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.076880 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.078367 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.078820 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.078970 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.081573 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:07.081579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:07.081582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:07.081624 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:07.084148 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.084920 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.084931 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.084933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.085062 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.085539 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.085545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.085548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.085901 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:07.086170 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.086520 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.086770 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 18:48:07.091797 13295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:48:07.099020 13295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:48:07.140437 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.112317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46428)
0927 18:48:07.113788 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.113844 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:07.114086 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:07.132223 (+ 18137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:07.132230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:07.132264 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:07.132879 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.132898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.134850 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.134855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.137489 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.137508 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.139290 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.139295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.139516 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.139595 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 18:48:07.139708 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:48:07.139934 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":2,"thread_start_us":1488,"threads_started":1}
I20260927 18:48:07.140743 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.112838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.31.26.193:34073)
0927 18:48:07.131512 (+ 18674us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.131522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.131549 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:48:07.131632 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.131637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.131936 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.132866 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:07.133165 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.134706 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.137691 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.139052 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.139890 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.140134 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.140218 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.140402 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18468,"thread_start_us":18321,"threads_started":1}
I20260927 18:48:07.141023 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.113559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.31.26.194:38785)
0927 18:48:07.130274 (+ 16715us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.130289 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.131224 (+ 935us) server_negotiation.cc:407] Connection header received
0927 18:48:07.131600 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.131609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.131936 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.132975 (+ 1039us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:07.133890 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.135518 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.137727 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.139275 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.139425 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.139644 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.140548 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.140722 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16411,"thread_start_us":16699,"threads_started":1}
I20260927 18:48:07.141763 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.115001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45089 (local address 127.0.0.1:37990)
0927 18:48:07.133300 (+ 18299us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.133318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:07.133461 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:07.134139 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:07.134144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:07.134169 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:07.134543 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.134555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.136145 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.136151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.138500 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.138517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.140864 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.140871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.141066 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.141137 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 18:48:07.141224 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:48:07.141368 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18165,"mutex_wait_us":1,"spinlock_wait_cycles":6400,"thread_start_us":17715,"threads_started":1}
I20260927 18:48:07.141973 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.115729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.31.26.195:45089)
0927 18:48:07.132466 (+ 16737us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.132478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.133513 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 18:48:07.133634 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.133643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.133900 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.134134 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:07.134743 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.135978 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.138752 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.140264 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.140399 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.141329 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.141423 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.141595 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16537,"thread_start_us":16389,"threads_started":1}
I20260927 18:48:07.142329 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.113020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:38785 (local address 127.0.0.1:53236)
0927 18:48:07.131072 (+ 18052us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.131096 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:07.131369 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:07.133300 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:07.133304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:07.133337 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:07.133718 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.133727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.135692 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.135697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.137549 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.137563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.139802 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.139809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.139990 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.140024 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:48:07.140115 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:48:07.140267 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17910,"mutex_wait_us":21,"thread_start_us":17726,"threads_started":1}
I20260927 18:48:07.145413 13687 tablet_service.cc:1467] Processing CreateTablet for tablet baa949a4c515441c8b5c2fa4fdcaf463 (DEFAULT_TABLE table=test-table [id=feb9eaaef89d4583aa9c70fc0b9adfc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:07.145445 13419 tablet_service.cc:1467] Processing CreateTablet for tablet baa949a4c515441c8b5c2fa4fdcaf463 (DEFAULT_TABLE table=test-table [id=feb9eaaef89d4583aa9c70fc0b9adfc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:07.146067 13553 tablet_service.cc:1467] Processing CreateTablet for tablet baa949a4c515441c8b5c2fa4fdcaf463 (DEFAULT_TABLE table=test-table [id=feb9eaaef89d4583aa9c70fc0b9adfc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:07.147008 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa949a4c515441c8b5c2fa4fdcaf463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:07.147013 13419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa949a4c515441c8b5c2fa4fdcaf463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:07.147367 13553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa949a4c515441c8b5c2fa4fdcaf463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:07.161970 13771 tablet_bootstrap.cc:492] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Bootstrap starting.
I20260927 18:48:07.165570 13771 tablet_bootstrap.cc:654] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:07.167040 13771 log.cc:824] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:07.167829 13773 tablet_bootstrap.cc:492] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Bootstrap starting.
I20260927 18:48:07.168599 13772 tablet_bootstrap.cc:492] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Bootstrap starting.
I20260927 18:48:07.170984 13771 tablet_bootstrap.cc:492] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: No bootstrap required, opened a new log
I20260927 18:48:07.171290 13771 ts_tablet_manager.cc:1397] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:48:07.172044 13773 tablet_bootstrap.cc:654] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:07.172281 13772 tablet_bootstrap.cc:654] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:07.173246 13773 log.cc:824] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:07.173655 13772 log.cc:824] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:07.176168 13773 tablet_bootstrap.cc:492] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: No bootstrap required, opened a new log
I20260927 18:48:07.176472 13773 ts_tablet_manager.cc:1397] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 18:48:07.180148 13772 tablet_bootstrap.cc:492] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: No bootstrap required, opened a new log
I20260927 18:48:07.180466 13772 ts_tablet_manager.cc:1397] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 18:48:07.183212 13771 raft_consensus.cc:348] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.183745 13771 raft_consensus.cc:374] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:07.183853 13771 raft_consensus.cc:729] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364356cf0b6149fc8aeea252bb2dca90, State: Initialized, Role: FOLLOWER
I20260927 18:48:07.184566 13771 consensus_queue.cc:260] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.185868 13752 heartbeater.cc:503] Master 127.31.26.254:44371 was elected leader, sending a full tablet report...
I20260927 18:48:07.185920 13773 raft_consensus.cc:348] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.186389 13773 raft_consensus.cc:374] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:07.186508 13773 raft_consensus.cc:729] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f1d42a45a94eb8ad2c1e8523b197ce, State: Initialized, Role: FOLLOWER
I20260927 18:48:07.186578 13771 ts_tablet_manager.cc:1428] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 18:48:07.187134 13773 consensus_queue.cc:260] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.189888 13618 heartbeater.cc:503] Master 127.31.26.254:44371 was elected leader, sending a full tablet report...
I20260927 18:48:07.190724 13773 ts_tablet_manager.cc:1428] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 18:48:07.192349 13772 raft_consensus.cc:348] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.192885 13772 raft_consensus.cc:374] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:07.192971 13772 raft_consensus.cc:729] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf449957ed34859877548b9345b0c1a, State: Initialized, Role: FOLLOWER
I20260927 18:48:07.193689 13772 consensus_queue.cc:260] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.195689 13772 ts_tablet_manager.cc:1428] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260927 18:48:07.232795 13619 tablet.cc:2369] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:48:07.285333 13753 tablet.cc:2369] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:07.307538 13779 raft_consensus.cc:482] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:48:07.307893 13779 raft_consensus.cc:504] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.309957 13779 leader_election.cc:288] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089), d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785)
I20260927 18:48:07.317996 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.310543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:55495 (local address 127.31.26.194:38785)
0927 18:48:07.310749 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.310756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.311554 (+ 798us) server_negotiation.cc:407] Connection header received
0927 18:48:07.311732 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.311737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.311820 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.311929 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:07.312891 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.314090 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.316067 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.316907 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.317061 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.317221 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.317585 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.317690 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:48:07.318117 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.310548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:38785 (local address 127.31.26.193:55495)
0927 18:48:07.311428 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.311446 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:07.311582 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:07.312000 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:07.312005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:07.312057 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:07.312624 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.312636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.314308 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.314313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.315833 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.315853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.317212 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.317219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.317400 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.317438 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:48:07.317555 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 18:48:07.317709 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":32,"spinlock_wait_cycles":5376,"thread_start_us":495,"threads_started":1}
I20260927 18:48:07.319271 13573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa949a4c515441c8b5c2fa4fdcaf463" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" is_pre_election: true
I20260927 18:48:07.319965 13573 raft_consensus.cc:2454] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 0.
I20260927 18:48:07.320037 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.310543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:37441 (local address 127.31.26.195:45089)
0927 18:48:07.310749 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.310755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.311555 (+ 800us) server_negotiation.cc:407] Connection header received
0927 18:48:07.311760 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.311765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.311836 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.311929 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:07.312891 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.314124 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.317856 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.318957 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.319173 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.319249 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.319574 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.319766 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:48:07.320114 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.310392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45089 (local address 127.31.26.193:37441)
0927 18:48:07.311389 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.311411 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:07.311624 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:07.312000 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:07.312005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:07.312051 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:07.312620 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.312636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.316031 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.316035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.317666 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:07.317680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.319190 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.319196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:07.319369 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.319425 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 18:48:07.319547 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 18:48:07.319694 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":27,"spinlock_wait_cycles":40192,"thread_start_us":556,"threads_started":1}
I20260927 18:48:07.320811 13373 leader_election.cc:302] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf449957ed34859877548b9345b0c1a, d5f1d42a45a94eb8ad2c1e8523b197ce; no voters:
I20260927 18:48:07.321259 13779 raft_consensus.cc:2781] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:48:07.321440 13779 raft_consensus.cc:482] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:48:07.321336 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa949a4c515441c8b5c2fa4fdcaf463" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364356cf0b6149fc8aeea252bb2dca90" is_pre_election: true
I20260927 18:48:07.321604 13779 raft_consensus.cc:3037] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:07.321933 13707 raft_consensus.cc:2454] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 0.
I20260927 18:48:07.324123 13779 raft_consensus.cc:504] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.324738 13779 leader_election.cc:288] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 election: Requested vote from peers 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089), d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785)
I20260927 18:48:07.325410 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa949a4c515441c8b5c2fa4fdcaf463" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364356cf0b6149fc8aeea252bb2dca90"
I20260927 18:48:07.325464 13573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa949a4c515441c8b5c2fa4fdcaf463" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
I20260927 18:48:07.325641 13707 raft_consensus.cc:3037] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:07.325690 13573 raft_consensus.cc:3037] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:07.327962 13707 raft_consensus.cc:2454] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 1.
I20260927 18:48:07.328079 13573 raft_consensus.cc:2454] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 1.
I20260927 18:48:07.328568 13372 leader_election.cc:302] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364356cf0b6149fc8aeea252bb2dca90, 6bf449957ed34859877548b9345b0c1a; no voters:
I20260927 18:48:07.328944 13779 raft_consensus.cc:2781] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:07.329519 13779 raft_consensus.cc:686] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 1 LEADER]: Becoming Leader. State: Replica: 6bf449957ed34859877548b9345b0c1a, State: Running, Role: LEADER
I20260927 18:48:07.330154 13779 consensus_queue.cc:237] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:07.335675 13294 catalog_manager.cc:5495] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bf449957ed34859877548b9345b0c1a (127.31.26.193). New cstate: current_term: 1 leader_uuid: "6bf449957ed34859877548b9345b0c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } health_report { overall_health: UNKNOWN } } }
I20260927 18:48:07.369973 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.363751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.31.26.193:34073)
0927 18:48:07.364024 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.364032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.364135 (+ 103us) server_negotiation.cc:407] Connection header received
0927 18:48:07.364402 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.364406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.364499 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.364681 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:07.365122 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.365848 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.367453 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.367883 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.368054 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.368695 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:07.369329 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:48:07.369511 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.369592 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.369686 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 18:48:07.379384 13707 raft_consensus.cc:1261] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Refusing update from remote peer 6bf449957ed34859877548b9345b0c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:07.379384 13573 raft_consensus.cc:1261] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Refusing update from remote peer 6bf449957ed34859877548b9345b0c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:07.380724 13782 consensus_queue.cc:1035] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:07.381198 13779 consensus_queue.cc:1035] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:07.392652 13785 mvcc.cc:205] Tried to move back new op lower bound from 7334030898693709824 to 7334030898508894208. Current Snapshot: MvccSnapshot[applied={T|T < 7334030898693709824}]
I20260927 18:48:07.394186 13786 mvcc.cc:205] Tried to move back new op lower bound from 7334030898693709824 to 7334030898508894208. Current Snapshot: MvccSnapshot[applied={T|T < 7334030898693709824}]
I20260927 18:48:07.417448 13294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:07.417793 13294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:07.422397 13294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Sending DeleteTablet for 3 replicas of tablet baa949a4c515441c8b5c2fa4fdcaf463
I20260927 18:48:07.423945 13553 tablet_service.cc:1514] Processing DeleteTablet for tablet baa949a4c515441c8b5c2fa4fdcaf463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:07 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:53236
I20260927 18:48:07.423954 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet baa949a4c515441c8b5c2fa4fdcaf463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:07 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:37990
I20260927 18:48:07.426085 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13356
I20260927 18:48:07.437242 13795 tablet_replica.cc:330] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: stopping tablet replica
I20260927 18:48:07.437616 13419 tablet_service.cc:1514] Processing DeleteTablet for tablet baa949a4c515441c8b5c2fa4fdcaf463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:07 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:46428
I20260927 18:48:07.437776 13795 raft_consensus.cc:2229] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:07.438350 13795 raft_consensus.cc:2258] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:07.439602 13796 tablet_replica.cc:330] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: stopping tablet replica
I20260927 18:48:07.440054 13796 raft_consensus.cc:2229] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:07.440915 13796 raft_consensus.cc:2258] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:07.442720 13795 ts_tablet_manager.cc:1905] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:07.454495 13795 ts_tablet_manager.cc:1918] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:07.454763 13795 log.cc:1196] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/baa949a4c515441c8b5c2fa4fdcaf463
I20260927 18:48:07.454926 13796 ts_tablet_manager.cc:1905] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:07.455353 13795 ts_tablet_manager.cc:1939] T baa949a4c515441c8b5c2fa4fdcaf463 P 364356cf0b6149fc8aeea252bb2dca90: Deleting consensus metadata
I20260927 18:48:07.457669 13279 catalog_manager.cc:4841] TS 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089): tablet baa949a4c515441c8b5c2fa4fdcaf463 (table test-table [id=feb9eaaef89d4583aa9c70fc0b9adfc5]) successfully deleted
I20260927 18:48:07.463699 13796 ts_tablet_manager.cc:1918] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:07.463943 13796 log.cc:1196] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/baa949a4c515441c8b5c2fa4fdcaf463
I20260927 18:48:07.464617 13796 ts_tablet_manager.cc:1939] T baa949a4c515441c8b5c2fa4fdcaf463 P d5f1d42a45a94eb8ad2c1e8523b197ce: Deleting consensus metadata
I20260927 18:48:07.467267 13280 catalog_manager.cc:4841] TS d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785): tablet baa949a4c515441c8b5c2fa4fdcaf463 (table test-table [id=feb9eaaef89d4583aa9c70fc0b9adfc5]) successfully deleted
I20260927 18:48:07.469213 13797 tablet_replica.cc:330] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: stopping tablet replica
Sep 27 18:48:07 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:48:07.470202 13797 raft_consensus.cc:2229] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:48:07.471127 13797 raft_consensus.cc:2258] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:07.485811 13797 ts_tablet_manager.cc:1905] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:07.489183 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.438847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.31.26.193:34073)
0927 18:48:07.439065 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.439071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.439586 (+ 515us) server_negotiation.cc:407] Connection header received
0927 18:48:07.439952 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.439961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.440065 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.440148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:07.447211 (+ 7063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.448643 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.451015 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.451823 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.451964 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.472706 (+ 20742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:07.472723 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:07.472742 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:07.472802 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:07.484432 (+ 11630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.486185 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.486195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.486198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.486379 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.486909 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.486914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.486917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.487453 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:07.487763 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.488448 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.488732 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:48:07.490890 13479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:07.598148 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13356
W20260927 18:48:07.617712 13279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.26.193:34073 (error 108)
I20260927 18:48:07.618006 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13490
I20260927 18:48:07.619829 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.619273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46454)
0927 18:48:07.619508 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.619637 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 18:48:07.620157 13279 catalog_manager.cc:4584] TS 6bf449957ed34859877548b9345b0c1a (127.31.26.193:34073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet baa949a4c515441c8b5c2fa4fdcaf463: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Sep 27 18:48:07 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:48:07.635169 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.618926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.31.26.194:38785)
0927 18:48:07.619148 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.619154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.619176 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:48:07.619273 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.619279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.619369 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.619479 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:07.621167 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.621973 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.623358 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.623762 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.623859 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.628530 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:07.628551 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:07.628568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:07.628611 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:07.632225 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.633133 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.633148 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.633153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.633305 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.633747 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.633751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.633753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.634211 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:07.634487 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.634711 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.634906 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:48:07.636073 13613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:07.674274 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.673655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46470)
0927 18:48:07.673941 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.674065 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 18:48:07.675503 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.675066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46476)
0927 18:48:07.675240 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.675348 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:48:07.731494 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13490
I20260927 18:48:07.736199 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.735670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46480)
0927 18:48:07.735922 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.736021 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 18:48:07.737545 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.737051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46486)
0927 18:48:07.737305 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.737387 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 18:48:07.749780 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13624
Sep 27 18:48:07 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:48:07.768446 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.750702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.31.26.195:45089)
0927 18:48:07.750900 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:07.750905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:07.750926 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:48:07.751122 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:07.751126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:07.751194 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:07.751288 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:07.752771 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.753726 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.755227 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:07.755644 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:07.755762 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:07.761292 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:07.761312 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:07.761329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:07.761377 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:07.764558 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.765891 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.765901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.765904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.766083 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:07.766602 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:07.766613 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:07.766615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:07.767033 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:07.767389 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:07.767796 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 18:48:07.768057 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:48:07.769389 13747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:07.818629 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.818065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46490)
0927 18:48:07.818327 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.818427 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 18:48:07.820186 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:07.819544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46506)
0927 18:48:07.819907 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:07.820011 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260927 18:48:07.865252 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13624
I20260927 18:48:07.882555 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13263
I20260927 18:48:07.883880 13325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:07.999279 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13263
I20260927 18:48:08.022754 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44371
--webserver_interface=127.31.26.254
--webserver_port=46241
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:08.389712 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:08.390040 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:08.390195 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:08.397878 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:08.398026 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:08.398254 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:08.398403 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:08.398478 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:08.409453 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=46241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:08.410950 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:08.412475 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:08.438119 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:08.438159 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:08.438100 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:08.438977 13804 server_base.cc:1034] running on GCE node
I20260927 18:48:08.440025 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:08.441916 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:08.443184 13804 hybrid_clock.cc:648] HybridClock initialized: now 1790534888443138 us; error 44 us; skew 500 ppm
I20260927 18:48:08.443858 13804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:08 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534888, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:08.448446 13804 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:48:08.450798 13804 webserver.cc:460] Webserver started at http://127.31.26.254:46241/ using document root <none> and password file <none>
I20260927 18:48:08.451805 13804 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:08.451999 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:08.459636 13804 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:48:08.463641 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:08.465778 13804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:08.466060 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "d50af23098b24f6faffef23d08d1fde7"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "66ad46ffb2a24f6208267dcb6ed4e603"
server_key_iv: "b6e65a5ed748ced0676961439fe6ae2f"
server_key_version: "encryptionkey@0"
I20260927 18:48:08.467357 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:08.494797 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:08.500676 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:08.501215 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:08.523007 13804 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44371
I20260927 18:48:08.523041 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44371 every 8 connection(s)
I20260927 18:48:08.524848 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:08.529452 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13804
I20260927 18:48:08.530510 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:34073
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:44371
--webserver_port=39081
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:48:08.534713 13872 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:48:08.540122 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap starting.
I20260927 18:48:08.562337 13872 log.cc:824] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:08.577301 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:48:08.577927 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap complete.
I20260927 18:48:08.587985 13872 raft_consensus.cc:348] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:08.588946 13872 raft_consensus.cc:729] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50af23098b24f6faffef23d08d1fde7, State: Initialized, Role: FOLLOWER
I20260927 18:48:08.589804 13872 consensus_queue.cc:260] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:08.590047 13872 raft_consensus.cc:388] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:08.590215 13872 raft_consensus.cc:482] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:08.590426 13872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:48:08.595005 13872 raft_consensus.cc:504] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:08.595868 13872 leader_election.cc:302] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d50af23098b24f6faffef23d08d1fde7; no voters:
I20260927 18:48:08.596611 13872 leader_election.cc:288] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:48:08.596772 13876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:48:08.598191 13876 raft_consensus.cc:686] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 LEADER]: Becoming Leader. State: Replica: d50af23098b24f6faffef23d08d1fde7, State: Running, Role: LEADER
I20260927 18:48:08.598872 13876 consensus_queue.cc:237] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:08.599339 13872 sys_catalog.cc:564] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:08.602074 13878 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50af23098b24f6faffef23d08d1fde7. Latest consensus state: current_term: 2 leader_uuid: "d50af23098b24f6faffef23d08d1fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } } }
I20260927 18:48:08.602592 13878 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:08.603195 13877 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d50af23098b24f6faffef23d08d1fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } } }
I20260927 18:48:08.603545 13877 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:08.609889 13882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:08.621145 13882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:08.623313 13882 catalog_manager.cc:1259] Loaded cluster ID: 197ccb8ad0dd4dbe9cd97d1c65804580
I20260927 18:48:08.623493 13882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:48:08.628248 13882 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 18:48:08.656303 13882 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 18:48:08.680212 13882 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:48:08.682713 13882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Loaded TSK: 0
I20260927 18:48:08.684964 13882 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:48:08.947383 13874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:08.947770 13874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:08.947942 13874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:08.955845 13874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:08.956037 13874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:08.956230 13874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:48:08.968024 13874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:34073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=39081
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:08.969615 13874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:08.971349 13874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:08.987694 13900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:08.987656 13901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:08.988327 13903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:08.988744 13874 server_base.cc:1034] running on GCE node
I20260927 18:48:08.989492 13874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:08.990650 13874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:08.991889 13874 hybrid_clock.cc:648] HybridClock initialized: now 1790534888991843 us; error 36 us; skew 500 ppm
I20260927 18:48:08.992254 13874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:08 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534888, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:08.995777 13874 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:48:08.997958 13874 webserver.cc:460] Webserver started at http://127.31.26.193:39081/ using document root <none> and password file <none>
I20260927 18:48:08.998764 13874 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:08.998920 13874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:09.005803 13874 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:09.009604 13910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:09.011363 13874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:48:09.011562 13874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "6bf449957ed34859877548b9345b0c1a"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "a79cc65c831e087cea422121325e7311"
server_key_iv: "519a84e97fb012d147a88dba14c167ff"
server_key_version: "encryptionkey@0"
I20260927 18:48:09.012678 13874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:09.032733 13874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:09.037881 13874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:09.038193 13874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:09.039402 13874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:09.041635 13917 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 18:48:09.047363 13917 ts_tablet_manager.cc:1852] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 18:48:09.048408 13917 ts_tablet_manager.cc:1905] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:09.056540 13917 ts_tablet_manager.cc:1918] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:09.056809 13917 log.cc:1196] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/baa949a4c515441c8b5c2fa4fdcaf463
I20260927 18:48:09.057421 13917 ts_tablet_manager.cc:1939] T baa949a4c515441c8b5c2fa4fdcaf463 P 6bf449957ed34859877548b9345b0c1a: Deleting consensus metadata
I20260927 18:48:09.057916 13874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 18:48:09.058053 13874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260927 18:48:09.058208 13874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:09.058305 13874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:09.108273 13874 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34073
I20260927 18:48:09.108341 14023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34073 every 8 connection(s)
I20260927 18:48:09.110489 13874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:48:09.120504 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 13874
I20260927 18:48:09.121373 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:38785
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:44371
--webserver_port=43235
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:48:09 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534888, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:09.142052 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:09.115099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:36961)
0927 18:48:09.116225 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:09.116295 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:09.119592 (+ 3297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:09.121990 (+ 2398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:09.122004 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:09.122990 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:09.124035 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:09.124076 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.126558 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.126564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:09.127924 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:09.127940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.129186 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.129194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:09.129507 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:09.130345 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:09.130372 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:09.133245 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:09.137160 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:09.137171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:09.137192 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:09.137701 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:09.138223 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:09.138229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:09.138233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:09.138437 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:09.139272 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:09.139282 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:09.139672 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 18:48:09.139960 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 18:48:09.140583 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":1177,"threads_started":1}
I20260927 18:48:09.142052 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:09.116236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:36961 (local address 127.31.26.254:44371)
0927 18:48:09.116804 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:09.116820 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:09.116871 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:48:09.120390 (+ 3519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:09.120415 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:09.120994 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:09.121520 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:09.124389 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.126336 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.128143 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.128856 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.129136 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:09.133605 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:09.133616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:09.133636 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:09.133682 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:09.136936 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:09.137873 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:09.137882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:09.137889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:09.138081 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:09.138588 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:09.138592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:09.138595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:09.139108 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:09.139335 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:09.140235 (+ 900us) server_negotiation.cc:299] Negotiation successful
0927 18:48:09.140663 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":1142,"threads_started":1}
I20260927 18:48:09.145299 14024 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44371
I20260927 18:48:09.145663 14024 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:09.146420 14024 heartbeater.cc:511] Master 127.31.26.254:44371 requested a full tablet report, sending...
I20260927 18:48:09.148741 13837 ts_manager.cc:194] Registered new tserver with Master: 6bf449957ed34859877548b9345b0c1a (127.31.26.193:34073)
I20260927 18:48:09.150754 13837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:36961
W20260927 18:48:09.481215 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:09.481608 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:09.481796 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:09.489807 14028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:09.490005 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:09.490206 14028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:48:09.501271 14028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:38785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=43235
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:09.502882 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:09.504290 14028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:09.519745 14036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:09.519838 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:09.519838 14034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:09.520646 14028 server_base.cc:1034] running on GCE node
I20260927 18:48:09.521456 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:09.522687 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:09.523965 14028 hybrid_clock.cc:648] HybridClock initialized: now 1790534889523885 us; error 68 us; skew 500 ppm
I20260927 18:48:09.524394 14028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:09 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534889, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:09.529057 14028 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:48:09.531244 14028 webserver.cc:460] Webserver started at http://127.31.26.194:43235/ using document root <none> and password file <none>
I20260927 18:48:09.532234 14028 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:09.532413 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:09.540300 14028 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:48:09.544461 14043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:09.546470 14028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 18:48:09.546707 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "829f46b3dfd86f421240391638734f3b"
server_key_iv: "9a978fcc636d16cee8c91f140ac9b512"
server_key_version: "encryptionkey@0"
I20260927 18:48:09.547786 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:09.564400 14028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:09.569674 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:09.569950 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:09.571080 14028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:09.572727 14028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:09.572822 14028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:09.572902 14028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:09.572962 14028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:09.609709 14028 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:38785
I20260927 18:48:09.609800 14155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:38785 every 8 connection(s)
I20260927 18:48:09.611384 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:48:09.618752 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14028
I20260927 18:48:09.619443 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:45089
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:44371
--webserver_port=41841
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:48:09 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534889, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:09.637425 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:09.614486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:37803 (local address 127.31.26.254:44371)
0927 18:48:09.614695 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:09.614700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:09.616097 (+ 1397us) server_negotiation.cc:407] Connection header received
0927 18:48:09.618129 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:09.618135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:09.618286 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:09.618403 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:09.620430 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.621821 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.623676 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.624467 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.624629 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:09.630004 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:09.630010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:09.630014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:09.630066 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:09.633019 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:09.634346 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:09.634354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:09.634358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:09.634554 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:09.635175 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:09.635182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:09.635185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:09.635560 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:09.635699 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:09.636807 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0927 18:48:09.637054 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:48:09.639191 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:09.614983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:37803)
0927 18:48:09.615885 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:09.615937 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:09.617807 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:09.618727 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:09.618736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:09.619404 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:09.620191 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:09.620212 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.622091 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.622095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:09.623469 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:09.623486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:09.624763 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:09.624768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:09.625032 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:09.626016 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:09.626042 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:09.629658 (+ 3616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:09.633297 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:09.633312 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:09.633336 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:09.634143 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:09.634710 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:09.634718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:09.634725 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:09.635028 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:09.635726 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:09.635738 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:09.636292 (+ 554us) client_negotiation.cc:769] Sending connection context
0927 18:48:09.636669 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 18:48:09.637397 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":437,"threads_started":1}
I20260927 18:48:09.641268 14156 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44371
I20260927 18:48:09.641672 14156 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:09.642535 14156 heartbeater.cc:511] Master 127.31.26.254:44371 requested a full tablet report, sending...
I20260927 18:48:09.644438 13837 ts_manager.cc:194] Registered new tserver with Master: d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785)
I20260927 18:48:09.645399 13837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:37803
W20260927 18:48:09.993067 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:09.993503 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:09.993693 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:10.001888 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:10.002084 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:10.002271 14159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:48:10.012709 14159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:45089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=41841
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:10.014364 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:10.015849 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:10.030637 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:10.030584 14167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:10.030610 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:10.031275 14159 server_base.cc:1034] running on GCE node
I20260927 18:48:10.032083 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:10.033336 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:10.034641 14159 hybrid_clock.cc:648] HybridClock initialized: now 1790534890034577 us; error 55 us; skew 500 ppm
I20260927 18:48:10.035107 14159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:10 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:10.039059 14159 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:48:10.040930 14159 webserver.cc:460] Webserver started at http://127.31.26.195:41841/ using document root <none> and password file <none>
I20260927 18:48:10.041795 14159 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:10.041970 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:10.048067 14159 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:48:10.051523 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:10.053381 14159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:10.053587 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "364356cf0b6149fc8aeea252bb2dca90"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "ba6a25c7339bf4b13a5090efe383680f"
server_key_iv: "a269a9524911c4d1c0288d3f764387a2"
server_key_version: "encryptionkey@0"
I20260927 18:48:10.054704 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:10.068151 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:10.073405 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:10.073717 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:10.074847 14159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:10.076402 14159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:10.076509 14159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:10.076635 14159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:10.076714 14159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:10.127135 14159 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:45089
I20260927 18:48:10.127195 14286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:45089 every 8 connection(s)
I20260927 18:48:10.128988 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:48:10.131880 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14159
Sep 27 18:48:10 dist-test-slave-rfd5 krb5kdc[13250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:10.153350 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.132916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.31.26.254:44371)
0927 18:48:10.133513 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.133522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.137740 (+ 4218us) server_negotiation.cc:407] Connection header received
0927 18:48:10.137972 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.137978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.138053 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.138172 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:10.139828 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.140722 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.142271 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.142774 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.142938 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.146023 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:10.146028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:10.146032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:10.146076 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:10.149889 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.150858 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.150864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.150867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.151018 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.151487 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.151514 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.151520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.151991 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:10.152244 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.152695 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.152947 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"spinlock_wait_cycles":1280,"thread_start_us":296,"threads_started":1}
I20260927 18:48:10.154320 14024 heartbeater.cc:503] Master 127.31.26.254:44371 was elected leader, sending a full tablet report...
I20260927 18:48:10.155328 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.132245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:49259 (local address 127.31.26.254:44371)
0927 18:48:10.132435 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.132440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.134352 (+ 1912us) server_negotiation.cc:407] Connection header received
0927 18:48:10.137892 (+ 3540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.137897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.138004 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.138128 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:10.139783 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.140763 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.142487 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.142991 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.143128 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.146960 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:10.146970 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:10.146981 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:10.147023 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:10.150491 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.151563 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.151566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.151569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.151744 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.152446 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.152451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.152454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.152890 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:10.153060 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.154209 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.154950 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":5376}
I20260927 18:48:10.156330 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.132937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:49259)
0927 18:48:10.134101 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:10.134210 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:10.137549 (+ 3339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:10.138402 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:10.138411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:10.138960 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:10.139581 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.139600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.140941 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.140946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.142323 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.142335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.143242 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.143250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.143537 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.144251 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:10.144275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:10.146651 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:10.150681 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:10.150693 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:10.150713 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:10.151434 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:10.151893 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:10.151896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:10.151899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:10.152272 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:10.153193 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:10.153199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:10.153666 (+ 467us) client_negotiation.cc:769] Sending connection context
0927 18:48:10.154068 (+ 402us) client_negotiation.cc:240] Negotiation successful
0927 18:48:10.154720 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":490,"threads_started":1}
I20260927 18:48:10.157891 14287 heartbeater.cc:347] Connected to a master server at 127.31.26.254:44371
I20260927 18:48:10.158200 14287 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:10.159029 14287 heartbeater.cc:511] Master 127.31.26.254:44371 requested a full tablet report, sending...
I20260927 18:48:10.160633 13836 ts_manager.cc:194] Registered new tserver with Master: 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089)
I20260927 18:48:10.161615 13836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:49259
I20260927 18:48:10.170331 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:48:10.177745 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.171396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.31.26.254:44371)
0927 18:48:10.171633 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.171643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.172045 (+ 402us) server_negotiation.cc:407] Connection header received
0927 18:48:10.172230 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.172234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.172377 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.172621 (+ 244us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:10.173073 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.173937 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.175431 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.175860 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.176017 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.176556 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:10.177075 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:48:10.177323 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.177375 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.177471 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:48:10.178721 13836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:48:10.186364 13836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:48:10.221264 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.201869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.31.26.195:45089)
0927 18:48:10.211406 (+ 9537us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.211420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.212224 (+ 804us) server_negotiation.cc:407] Connection header received
0927 18:48:10.212343 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.212353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.212663 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.212896 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:10.214239 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.215811 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.218692 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.219958 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.220149 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.220411 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.220716 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.220884 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9320,"thread_start_us":9151,"threads_started":1}
I20260927 18:48:10.221560 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.199352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34073 (local address 127.0.0.1:46526)
0927 18:48:10.200462 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:10.200506 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:10.200723 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:10.213251 (+ 12528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:10.213260 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:10.213304 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:10.214203 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.214215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.215874 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.215879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.218519 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.218536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.220600 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.220608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.220779 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.220812 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:48:10.220907 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:48:10.221067 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":318,"spinlock_wait_cycles":1792,"thread_start_us":839,"threads_started":1}
I20260927 18:48:10.221621 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.201195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45089 (local address 127.0.0.1:37998)
0927 18:48:10.211907 (+ 10712us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:10.211931 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:10.212084 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:10.213251 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:10.213260 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:10.213310 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:10.213943 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.213962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.216040 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.216049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.218476 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.218490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.220198 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.220207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.220419 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.220479 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:48:10.220630 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:48:10.220846 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10574,"mutex_wait_us":186,"spinlock_wait_cycles":5248,"thread_start_us":10424,"threads_started":1}
I20260927 18:48:10.222409 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.199633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.31.26.193:34073)
0927 18:48:10.212254 (+ 12621us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.212264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.212288 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:48:10.212368 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.212375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.212662 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.212896 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:10.214392 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.215727 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.218743 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.220161 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.220309 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.220554 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.221844 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.222038 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12405,"thread_start_us":13692,"threads_started":1}
I20260927 18:48:10.222589 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.200362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:38785 (local address 127.0.0.1:53254)
0927 18:48:10.210260 (+ 9898us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:10.210290 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:10.210461 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:10.214661 (+ 4200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:10.214666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:10.214693 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:10.215058 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.215070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.217035 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.217041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.219472 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.219490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.221758 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.221766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.221938 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.221974 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:48:10.222062 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:48:10.222201 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9691,"mutex_wait_us":1,"thread_start_us":10219,"threads_started":1}
I20260927 18:48:10.223470 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.200993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.31.26.194:38785)
0927 18:48:10.213923 (+ 12930us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.213929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.213956 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:48:10.214034 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.214040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.214400 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.214664 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:10.215263 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.216857 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.219676 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.221237 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.221371 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.221585 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.222961 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.223149 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12769,"thread_start_us":12626,"threads_started":1}
I20260927 18:48:10.224473 13959 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4c73923c154fc5b035308bee6d5fda (DEFAULT_TABLE table=test-table [id=5b1fc9ebeba141efb74c3ecf20c2eb35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:10.226028 13959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4c73923c154fc5b035308bee6d5fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:10.226166 14222 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4c73923c154fc5b035308bee6d5fda (DEFAULT_TABLE table=test-table [id=5b1fc9ebeba141efb74c3ecf20c2eb35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:10.227133 14091 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4c73923c154fc5b035308bee6d5fda (DEFAULT_TABLE table=test-table [id=5b1fc9ebeba141efb74c3ecf20c2eb35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:10.227586 14222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4c73923c154fc5b035308bee6d5fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:10.228127 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4c73923c154fc5b035308bee6d5fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:10.236826 14303 tablet_bootstrap.cc:492] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Bootstrap starting.
I20260927 18:48:10.240911 14303 tablet_bootstrap.cc:654] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:10.242449 14303 log.cc:824] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:10.243001 14304 tablet_bootstrap.cc:492] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Bootstrap starting.
I20260927 18:48:10.245887 14305 tablet_bootstrap.cc:492] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Bootstrap starting.
I20260927 18:48:10.246531 14304 tablet_bootstrap.cc:654] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:10.247973 14304 log.cc:824] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:10.249935 14305 tablet_bootstrap.cc:654] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:10.251086 14305 log.cc:824] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:10.251660 14303 tablet_bootstrap.cc:492] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: No bootstrap required, opened a new log
I20260927 18:48:10.251660 14304 tablet_bootstrap.cc:492] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: No bootstrap required, opened a new log
I20260927 18:48:10.251965 14304 ts_tablet_manager.cc:1397] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 18:48:10.251965 14303 ts_tablet_manager.cc:1397] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260927 18:48:10.256580 14305 tablet_bootstrap.cc:492] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: No bootstrap required, opened a new log
I20260927 18:48:10.256918 14305 ts_tablet_manager.cc:1397] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:48:10.263711 14304 raft_consensus.cc:348] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.263710 14303 raft_consensus.cc:348] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.264271 14304 raft_consensus.cc:374] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:10.264271 14303 raft_consensus.cc:374] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:10.264387 14303 raft_consensus.cc:729] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf449957ed34859877548b9345b0c1a, State: Initialized, Role: FOLLOWER
I20260927 18:48:10.264387 14304 raft_consensus.cc:729] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364356cf0b6149fc8aeea252bb2dca90, State: Initialized, Role: FOLLOWER
I20260927 18:48:10.264920 14305 raft_consensus.cc:348] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.265149 14304 consensus_queue.cc:260] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.265166 14303 consensus_queue.cc:260] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.265424 14305 raft_consensus.cc:374] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:10.265532 14305 raft_consensus.cc:729] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f1d42a45a94eb8ad2c1e8523b197ce, State: Initialized, Role: FOLLOWER
I20260927 18:48:10.266269 14305 consensus_queue.cc:260] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.268368 14156 heartbeater.cc:503] Master 127.31.26.254:44371 was elected leader, sending a full tablet report...
I20260927 18:48:10.269511 14287 heartbeater.cc:503] Master 127.31.26.254:44371 was elected leader, sending a full tablet report...
I20260927 18:48:10.270078 14303 ts_tablet_manager.cc:1428] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
I20260927 18:48:10.272019 14304 ts_tablet_manager.cc:1428] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260927 18:48:10.272202 14305 ts_tablet_manager.cc:1428] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260927 18:48:10.367308 14157 tablet.cc:2369] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:48:10.375712 14025 tablet.cc:2369] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:48:10.380891 14288 tablet.cc:2369] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:10.549957 14310 raft_consensus.cc:482] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:48:10.550303 14310 raft_consensus.cc:504] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.552395 14310 leader_election.cc:288] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089), d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785)
I20260927 18:48:10.562616 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.552936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:58105 (local address 127.31.26.194:38785)
0927 18:48:10.553191 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.553197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.554109 (+ 912us) server_negotiation.cc:407] Connection header received
0927 18:48:10.554344 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.554349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.554428 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.554559 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:10.555513 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.556610 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.559960 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.561307 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.561469 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.561549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.562185 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.562327 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:48:10.562594 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.552996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:35621 (local address 127.31.26.195:45089)
0927 18:48:10.553190 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.553197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.554050 (+ 853us) server_negotiation.cc:407] Connection header received
0927 18:48:10.554339 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.554345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.554433 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.554546 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:10.555486 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.556629 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.559970 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.561294 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.561469 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.561549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.562185 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.562327 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:48:10.562891 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.552834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:45089 (local address 127.31.26.193:35621)
0927 18:48:10.553890 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:10.553917 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:10.554100 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:10.554599 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:10.554605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:10.554657 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:10.555273 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.555295 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.556793 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.556800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.559724 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.559744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.561638 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.561644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.561912 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.562023 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 18:48:10.562130 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:48:10.562312 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":360,"spinlock_wait_cycles":16768,"thread_start_us":640,"threads_started":1}
I20260927 18:48:10.563290 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.552851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:38785 (local address 127.31.26.193:58105)
0927 18:48:10.553924 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:10.553939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:10.554165 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:10.554599 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:10.554605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:10.554657 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:10.555306 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.555315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.556793 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.556800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.559732 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:10.559744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.561620 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.561628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:10.561913 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.562023 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 18:48:10.562130 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:48:10.562312 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":347,"spinlock_wait_cycles":54016,"thread_start_us":529,"threads_started":1}
I20260927 18:48:10.564056 14242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4c73923c154fc5b035308bee6d5fda" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364356cf0b6149fc8aeea252bb2dca90" is_pre_election: true
I20260927 18:48:10.564571 14242 raft_consensus.cc:2454] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 0.
I20260927 18:48:10.564486 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4c73923c154fc5b035308bee6d5fda" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" is_pre_election: true
I20260927 18:48:10.564986 14111 raft_consensus.cc:2454] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 0.
I20260927 18:48:10.565354 13911 leader_election.cc:302] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364356cf0b6149fc8aeea252bb2dca90, 6bf449957ed34859877548b9345b0c1a; no voters:
I20260927 18:48:10.565828 14310 raft_consensus.cc:2781] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:48:10.565987 14310 raft_consensus.cc:482] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:48:10.566094 14310 raft_consensus.cc:3037] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:10.568261 14310 raft_consensus.cc:504] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.568943 14310 leader_election.cc:288] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 election: Requested vote from peers 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089), d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785)
I20260927 18:48:10.569496 14242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4c73923c154fc5b035308bee6d5fda" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364356cf0b6149fc8aeea252bb2dca90"
I20260927 18:48:10.569592 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4c73923c154fc5b035308bee6d5fda" candidate_uuid: "6bf449957ed34859877548b9345b0c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
I20260927 18:48:10.569778 14242 raft_consensus.cc:3037] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:10.569824 14111 raft_consensus.cc:3037] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:10.572561 14111 raft_consensus.cc:2454] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 1.
I20260927 18:48:10.572706 14242 raft_consensus.cc:2454] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf449957ed34859877548b9345b0c1a in term 1.
I20260927 18:48:10.573110 13912 leader_election.cc:302] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf449957ed34859877548b9345b0c1a, d5f1d42a45a94eb8ad2c1e8523b197ce; no voters:
I20260927 18:48:10.573587 14310 raft_consensus.cc:2781] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:10.574223 14310 raft_consensus.cc:686] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 1 LEADER]: Becoming Leader. State: Replica: 6bf449957ed34859877548b9345b0c1a, State: Running, Role: LEADER
I20260927 18:48:10.574748 14310 consensus_queue.cc:237] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } }
I20260927 18:48:10.580039 13836 catalog_manager.cc:5495] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bf449957ed34859877548b9345b0c1a (127.31.26.193). New cstate: current_term: 1 leader_uuid: "6bf449957ed34859877548b9345b0c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf449957ed34859877548b9345b0c1a" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 } health_report { overall_health: UNKNOWN } } }
I20260927 18:48:10.674871 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.668937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.31.26.193:34073)
0927 18:48:10.669203 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.669211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.669255 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:48:10.669598 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.669602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.669683 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.669856 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:10.670260 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.671004 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.672469 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.672876 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.673032 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.673647 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:10.674219 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:48:10.674406 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.674479 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.674587 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:48:10.684700 14242 raft_consensus.cc:1261] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Refusing update from remote peer 6bf449957ed34859877548b9345b0c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:10.684700 14111 raft_consensus.cc:1261] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Refusing update from remote peer 6bf449957ed34859877548b9345b0c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:10.686229 14310 consensus_queue.cc:1035] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 38785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:10.686587 14314 consensus_queue.cc:1035] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "364356cf0b6149fc8aeea252bb2dca90" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:10.699723 14316 mvcc.cc:205] Tried to move back new op lower bound from 7334030912231669760 to 7334030911797800960. Current Snapshot: MvccSnapshot[applied={T|T < 7334030912231669760}]
I20260927 18:48:10.700963 14317 mvcc.cc:205] Tried to move back new op lower bound from 7334030912231669760 to 7334030911797800960. Current Snapshot: MvccSnapshot[applied={T|T < 7334030912231669760}]
I20260927 18:48:10.719684 13836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:10.720002 13836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:10.724138 13836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Sending DeleteTablet for 3 replicas of tablet 2a4c73923c154fc5b035308bee6d5fda
I20260927 18:48:10.725618 14222 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4c73923c154fc5b035308bee6d5fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:10 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:37998
I20260927 18:48:10.725831 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13874
I20260927 18:48:10.725833 14091 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4c73923c154fc5b035308bee6d5fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:10 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:53254
I20260927 18:48:10.726776 13959 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4c73923c154fc5b035308bee6d5fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:10 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:46526
I20260927 18:48:10.738641 14329 tablet_replica.cc:330] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: stopping tablet replica
I20260927 18:48:10.739399 14326 tablet_replica.cc:330] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: stopping tablet replica
I20260927 18:48:10.739951 14326 raft_consensus.cc:2229] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:10.739971 14329 raft_consensus.cc:2229] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:48:10.740648 14326 raft_consensus.cc:2258] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:10.740823 14329 raft_consensus.cc:2258] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:10.741575 14327 tablet_replica.cc:330] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: stopping tablet replica
I20260927 18:48:10.742075 14327 raft_consensus.cc:2229] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:10.742662 14327 raft_consensus.cc:2258] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:10.743420 14326 ts_tablet_manager.cc:1905] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:10.745018 14327 ts_tablet_manager.cc:1905] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:10.745541 14329 ts_tablet_manager.cc:1905] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:10.753149 14326 ts_tablet_manager.cc:1918] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:10.753443 14326 log.cc:1196] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/2a4c73923c154fc5b035308bee6d5fda
I20260927 18:48:10.753906 14327 ts_tablet_manager.cc:1918] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:10.754174 14327 log.cc:1196] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/2a4c73923c154fc5b035308bee6d5fda
I20260927 18:48:10.754346 14326 ts_tablet_manager.cc:1939] T 2a4c73923c154fc5b035308bee6d5fda P 364356cf0b6149fc8aeea252bb2dca90: Deleting consensus metadata
I20260927 18:48:10.754854 14327 ts_tablet_manager.cc:1939] T 2a4c73923c154fc5b035308bee6d5fda P d5f1d42a45a94eb8ad2c1e8523b197ce: Deleting consensus metadata
I20260927 18:48:10.756642 13821 catalog_manager.cc:4841] TS 364356cf0b6149fc8aeea252bb2dca90 (127.31.26.195:45089): tablet 2a4c73923c154fc5b035308bee6d5fda (table test-table [id=5b1fc9ebeba141efb74c3ecf20c2eb35]) successfully deleted
I20260927 18:48:10.757295 13822 catalog_manager.cc:4841] TS d5f1d42a45a94eb8ad2c1e8523b197ce (127.31.26.194:38785): tablet 2a4c73923c154fc5b035308bee6d5fda (table test-table [id=5b1fc9ebeba141efb74c3ecf20c2eb35]) successfully deleted
I20260927 18:48:10.759500 14329 ts_tablet_manager.cc:1918] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:10.759696 14329 log.cc:1196] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/2a4c73923c154fc5b035308bee6d5fda
I20260927 18:48:10.760403 14329 ts_tablet_manager.cc:1939] T 2a4c73923c154fc5b035308bee6d5fda P 6bf449957ed34859877548b9345b0c1a: Deleting consensus metadata
I20260927 18:48:10.762777 13821 catalog_manager.cc:4841] TS 6bf449957ed34859877548b9345b0c1a (127.31.26.193:34073): tablet 2a4c73923c154fc5b035308bee6d5fda (table test-table [id=5b1fc9ebeba141efb74c3ecf20c2eb35]) successfully deleted
I20260927 18:48:10.764420 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.727591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.31.26.193:34073)
0927 18:48:10.727845 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.727856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.738971 (+ 11115us) server_negotiation.cc:407] Connection header received
0927 18:48:10.739248 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.739254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.739346 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.739475 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:10.741505 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.742647 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.744728 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.745448 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.745606 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.755779 (+ 10173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:10.755806 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:10.755825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:10.755890 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:10.760750 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.761776 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.761785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.761787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.761980 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.762463 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.762468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.762491 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.763017 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:10.763371 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.763772 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.764056 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:48:10.765282 14019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:10.870500 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13874
I20260927 18:48:10.889501 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14028
I20260927 18:48:10.904837 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:10.890313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.31.26.194:38785)
0927 18:48:10.890594 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:10.890600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:10.890621 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:48:10.890749 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:10.890754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:10.890818 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:10.890954 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:10.892432 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.893279 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.894642 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:10.895079 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:10.895240 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:10.898374 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:10.898390 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:10.898409 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:10.898451 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:10.901624 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.902454 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.902463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.902465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.902623 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:10.903117 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:10.903125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:10.903129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:10.903540 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:10.903860 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:10.904207 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 18:48:10.904451 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260927 18:48:10.905687 14151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:10.997835 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14028
I20260927 18:48:11.015444 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14159
I20260927 18:48:11.029718 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:11.016364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.31.26.195:45089)
0927 18:48:11.016583 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:11.016587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:11.016606 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:48:11.016663 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:11.016667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:11.016740 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:11.016827 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:11.018307 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:11.019056 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:11.020389 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:11.020785 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:11.020890 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:11.023746 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:11.023761 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:11.023778 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:11.023824 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:11.026869 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:11.027601 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:11.027610 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:11.027618 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:11.027775 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:11.028205 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:11.028209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:11.028212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:11.028640 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:11.028869 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:11.029233 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:48:11.029407 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 18:48:11.030421 14282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:11.120970 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14159
I20260927 18:48:11.138423 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 13804
I20260927 18:48:11.139809 13867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:11.256584 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 13804
I20260927 18:48:11.280311 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44371
--webserver_interface=127.31.26.254
--webserver_port=46241
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:11.634725 14334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:11.635052 14334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:11.635146 14334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:11.643327 14334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:11.643465 14334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:11.643515 14334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:11.643568 14334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:11.643605 14334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:11.654317 14334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=46241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:11.655828 14334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:11.657271 14334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:11.670675 14339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:11.670775 14340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:11.670905 14342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:11.672184 14334 server_base.cc:1034] running on GCE node
I20260927 18:48:11.673276 14334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:11.675010 14334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:11.676301 14334 hybrid_clock.cc:648] HybridClock initialized: now 1790534891676243 us; error 53 us; skew 500 ppm
I20260927 18:48:11.676723 14334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:11 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534891, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:11.680256 14334 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:48:11.682145 14334 webserver.cc:460] Webserver started at http://127.31.26.254:46241/ using document root <none> and password file <none>
I20260927 18:48:11.682950 14334 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:11.683122 14334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:11.689918 14334 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 18:48:11.696158 14349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:11.698076 14334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:11.698342 14334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "d50af23098b24f6faffef23d08d1fde7"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "66ad46ffb2a24f6208267dcb6ed4e603"
server_key_iv: "b6e65a5ed748ced0676961439fe6ae2f"
server_key_version: "encryptionkey@0"
I20260927 18:48:11.699679 14334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:11.716898 14334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:11.722267 14334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:11.722663 14334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:11.743463 14334 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44371
I20260927 18:48:11.743489 14400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44371 every 8 connection(s)
I20260927 18:48:11.745153 14334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:11.745814 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14334
I20260927 18:48:11.746974 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:34073
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:44371
--webserver_port=39081
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:48:11.756043 14401 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:48:11.759711 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap starting.
I20260927 18:48:11.781920 14401 log.cc:824] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:11.798988 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:48:11.799610 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7: Bootstrap complete.
I20260927 18:48:11.810173 14401 raft_consensus.cc:348] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:11.811270 14401 raft_consensus.cc:729] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50af23098b24f6faffef23d08d1fde7, State: Initialized, Role: FOLLOWER
I20260927 18:48:11.812070 14401 consensus_queue.cc:260] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:11.812393 14401 raft_consensus.cc:388] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:11.812561 14401 raft_consensus.cc:482] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:11.812763 14401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:48:11.817273 14401 raft_consensus.cc:504] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:11.817878 14401 leader_election.cc:302] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d50af23098b24f6faffef23d08d1fde7; no voters:
I20260927 18:48:11.818655 14401 leader_election.cc:288] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:48:11.818815 14405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:48:11.820258 14405 raft_consensus.cc:686] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [term 3 LEADER]: Becoming Leader. State: Replica: d50af23098b24f6faffef23d08d1fde7, State: Running, Role: LEADER
I20260927 18:48:11.820943 14405 consensus_queue.cc:237] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } }
I20260927 18:48:11.821352 14401 sys_catalog.cc:564] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:11.823820 14407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50af23098b24f6faffef23d08d1fde7. Latest consensus state: current_term: 3 leader_uuid: "d50af23098b24f6faffef23d08d1fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } } }
I20260927 18:48:11.824280 14407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:11.824994 14406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d50af23098b24f6faffef23d08d1fde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50af23098b24f6faffef23d08d1fde7" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44371 } } }
I20260927 18:48:11.825387 14406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50af23098b24f6faffef23d08d1fde7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:11.836309 14413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:11.850653 14413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:11.852512 14413 catalog_manager.cc:1259] Loaded cluster ID: 197ccb8ad0dd4dbe9cd97d1c65804580
I20260927 18:48:11.852660 14413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 18:48:11.855119 14413 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f84782289ad google::LogMessage::Fail() at ??:0
@ 0x7f847822b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f84782284de google::LogMessage::Flush() at ??:0
@ 0x7f8478229a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f848dc5b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f848dc58976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f848dce0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f848dce0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f847ae74345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f847ae7791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f847ae77799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f847ae52eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f847e12c6db start_thread at ??:0
@ 0x7f84770c261f clone at ??:0
W20260927 18:48:12.143509 14403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:12.143872 14403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:12.144053 14403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:12.151773 14403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:12.151970 14403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:12.152173 14403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:48:12.162703 14403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:34073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=39081
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:12.164127 14403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:12.165676 14403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:12.180100 14442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:12.180006 14443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:12.180039 14445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:12.180943 14403 server_base.cc:1034] running on GCE node
I20260927 18:48:12.181705 14403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:12.182731 14403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:12.183952 14403 hybrid_clock.cc:648] HybridClock initialized: now 1790534892183875 us; error 68 us; skew 500 ppm
I20260927 18:48:12.184366 14403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:12 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534892, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:12.187983 14403 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:48:12.189738 14403 webserver.cc:460] Webserver started at http://127.31.26.193:39081/ using document root <none> and password file <none>
I20260927 18:48:12.190480 14403 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:12.190599 14403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:12.196691 14403 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 18:48:12.200132 14452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:12.202339 14403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:48:12.202610 14403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "6bf449957ed34859877548b9345b0c1a"
format_stamp: "Formatted at 2026-09-27 18:48:05 on dist-test-slave-rfd5"
server_key: "a79cc65c831e087cea422121325e7311"
server_key_iv: "519a84e97fb012d147a88dba14c167ff"
server_key_version: "encryptionkey@0"
I20260927 18:48:12.203790 14403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:12.227963 14403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:12.233150 14403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:12.233481 14403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:12.234628 14403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:12.236208 14403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:12.236351 14403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:12.236503 14403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:12.236611 14403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:12.272922 14403 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34073
I20260927 18:48:12.272989 14564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34073 every 8 connection(s)
I20260927 18:48:12.274812 14403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:48:12.277156 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14403
I20260927 18:48:12.278033 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:38785
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:44371
--webserver_port=43235
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:48:12.280974 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:12.278785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:58589)
0927 18:48:12.279695 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:12.279961 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":379,"threads_started":1}
W20260927 18:48:12.281951 14565 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:44371 (0 consecutive failures): Network error: Failed to ping master at 127.31.26.254:44371: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
I20260927 18:48:12.282996 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:12.282579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:55415)
0927 18:48:12.282778 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:12.282872 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:48:12.284045 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:12.283682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:53477)
0927 18:48:12.283840 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:12.283927 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:48:12.284363 14565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:48:12.634352 14568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:12.634689 14568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:12.634789 14568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:12.643056 14568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:12.643199 14568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:12.643321 14568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:48:12.653556 14568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:38785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=43235
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:12.654965 14568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:12.656332 14568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:12.671209 14573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:12.671185 14574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:12.671022 14576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:12.672184 14568 server_base.cc:1034] running on GCE node
I20260927 18:48:12.672995 14568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:12.674335 14568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:12.675632 14568 hybrid_clock.cc:648] HybridClock initialized: now 1790534892675556 us; error 76 us; skew 500 ppm
I20260927 18:48:12.676019 14568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:12 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534892, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:12.679714 14568 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:48:12.681386 14568 webserver.cc:460] Webserver started at http://127.31.26.194:43235/ using document root <none> and password file <none>
I20260927 18:48:12.682142 14568 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:12.682252 14568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:12.688432 14568 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:48:12.691931 14583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:12.694012 14568 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:12.694229 14568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "d5f1d42a45a94eb8ad2c1e8523b197ce"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "829f46b3dfd86f421240391638734f3b"
server_key_iv: "9a978fcc636d16cee8c91f140ac9b512"
server_key_version: "encryptionkey@0"
I20260927 18:48:12.695329 14568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:12.723323 14568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:12.728868 14568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:12.729241 14568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:12.730508 14568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:12.732026 14568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:12.732128 14568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:12.732293 14568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:12.732360 14568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:12.769739 14568 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:38785
I20260927 18:48:12.769836 14695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:38785 every 8 connection(s)
I20260927 18:48:12.771394 14568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:48:12.775183 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14568
I20260927 18:48:12.775861 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:45089
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:44371
--webserver_port=41841
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:48:12.777655 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:12.775069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:48955)
0927 18:48:12.776077 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:12.776433 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":436,"threads_started":1}
W20260927 18:48:12.778483 14696 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:44371 (0 consecutive failures): Network error: Failed to ping master at 127.31.26.254:44371: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
I20260927 18:48:12.779495 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:12.779146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:44361)
0927 18:48:12.779308 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:12.779382 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:48:12.780459 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:12.780123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:49451)
0927 18:48:12.780271 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:12.780346 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 18:48:12.780786 14696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:48:13.137969 14699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:13.138279 14699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:13.138458 14699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:13.146288 14699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:13.146423 14699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:13.146632 14699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:48:13.156921 14699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:45089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=41841
--tserver_master_addrs=127.31.26.254:44371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:13.158388 14699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:13.159864 14699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:13.175001 14707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:13.174954 14705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:13.175041 14704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:13.176059 14699 server_base.cc:1034] running on GCE node
I20260927 18:48:13.176899 14699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:13.178056 14699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:13.179308 14699 hybrid_clock.cc:648] HybridClock initialized: now 1790534893179228 us; error 72 us; skew 500 ppm
I20260927 18:48:13.179791 14699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:13 dist-test-slave-rfd5 krb5kdc[13250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:13.184103 14699 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:48:13.186074 14699 webserver.cc:460] Webserver started at http://127.31.26.195:41841/ using document root <none> and password file <none>
I20260927 18:48:13.186919 14699 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:13.187125 14699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:13.193794 14699 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:48:13.197377 14714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:13.199508 14699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:48:13.199832 14699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "364356cf0b6149fc8aeea252bb2dca90"
format_stamp: "Formatted at 2026-09-27 18:48:06 on dist-test-slave-rfd5"
server_key: "ba6a25c7339bf4b13a5090efe383680f"
server_key_iv: "a269a9524911c4d1c0288d3f764387a2"
server_key_version: "encryptionkey@0"
I20260927 18:48:13.201344 14699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:13.220647 14699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:13.226742 14699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:13.227200 14699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:13.228577 14699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:13.230471 14699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:13.230665 14699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:13.230840 14699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:13.230957 14699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:13.271220 14699 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:45089
I20260927 18:48:13.271375 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:45089 every 8 connection(s)
I20260927 18:48:13.272886 14699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:48:13.273669 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14699
I20260927 18:48:13.279832 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:13.277554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:54389)
0927 18:48:13.278482 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:13.278763 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":350,"threads_started":1}
W20260927 18:48:13.280740 14827 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:44371 (0 consecutive failures): Network error: Failed to ping master at 127.31.26.254:44371: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
I20260927 18:48:13.281929 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:13.281412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:33689)
0927 18:48:13.281722 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:13.281802 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 18:48:13.282968 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:13.282598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:52437)
0927 18:48:13.282759 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:13.282843 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 18:48:13.283273 14827 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:48:13.285809 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:13.284930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:47951)
0927 18:48:13.285454 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:13.285553 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260927 18:48:13.782315 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:13.781478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:34279)
0927 18:48:13.781970 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:13.782073 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260927 18:48:14.285028 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:14.284004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:52785)
0927 18:48:14.284562 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:14.284709 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":265,"threads_started":1}
I20260927 18:48:14.287624 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:14.286805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:45495)
0927 18:48:14.287361 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:14.287447 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":311,"threads_started":1}
I20260927 18:48:14.784206 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:14.783415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:38557)
0927 18:48:14.783940 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:14.784035 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260927 18:48:15.286936 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:15.286187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:32845)
0927 18:48:15.286675 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:15.286766 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":234,"threads_started":1}
I20260927 18:48:15.289510 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:15.288636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:45933)
0927 18:48:15.289234 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:15.289321 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":319,"threads_started":1}
I20260927 18:48:15.786230 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:15.785330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:53965)
0927 18:48:15.785873 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:15.785986 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":249,"threads_started":1}
I20260927 18:48:16.288851 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:16.288074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:40617)
0927 18:48:16.288577 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:16.288672 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260927 18:48:16.291217 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:16.290453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:33793)
0927 18:48:16.290967 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:16.291049 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260927 18:48:16.788321 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:16.787408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:35281)
0927 18:48:16.788002 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:16.788105 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":310,"threads_started":1}
I20260927 18:48:17.290843 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:17.289981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:47623)
0927 18:48:17.290512 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:17.290655 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
I20260927 18:48:17.293376 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:17.292375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:49687)
0927 18:48:17.293006 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:17.293183 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":336,"threads_started":1}
I20260927 18:48:17.790436 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:17.789550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:44211)
0927 18:48:17.790132 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:17.790236 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260927 18:48:18.292708 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:18.291930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:55289)
0927 18:48:18.292420 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:18.292507 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":236,"threads_started":1}
I20260927 18:48:18.295125 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:18.294359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:40691)
0927 18:48:18.294846 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:18.294949 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260927 18:48:18.792426 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:18.791542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:53749)
0927 18:48:18.792115 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:18.792250 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
I20260927 18:48:19.294571 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:19.293814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:46051)
0927 18:48:19.294279 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:19.294382 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":224,"threads_started":1}
I20260927 18:48:19.297042 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:19.296188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:49037)
0927 18:48:19.296753 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:19.296836 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
I20260927 18:48:19.794878 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:19.793662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:54263)
0927 18:48:19.794363 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:19.794505 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":289,"threads_started":1}
I20260927 18:48:20.296720 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:20.295819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:52387)
0927 18:48:20.296371 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:20.296466 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":254,"threads_started":1}
I20260927 18:48:20.298859 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:20.298089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:58585)
0927 18:48:20.298599 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:20.298693 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
I20260927 18:48:20.796968 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:20.796155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:55849)
0927 18:48:20.796707 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:20.796792 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":316,"threads_started":1}
I20260927 18:48:21.298624 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:21.297823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:37189)
0927 18:48:21.298316 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:21.298426 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":217,"threads_started":1}
I20260927 18:48:21.300616 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:21.299848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:56221)
0927 18:48:21.300329 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:21.300449 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260927 18:48:21.799096 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:21.798109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:41715)
0927 18:48:21.798732 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:21.798845 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":269,"threads_started":1}
I20260927 18:48:22.300603 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:22.299751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:56697)
0927 18:48:22.300248 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:22.300340 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260927 18:48:22.302475 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:22.301689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:43335)
0927 18:48:22.302243 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:22.302320 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":338,"threads_started":1}
I20260927 18:48:22.801066 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:22.800154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:56483)
0927 18:48:22.800753 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:22.800845 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260927 18:48:23.302740 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:23.301923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:52657)
0927 18:48:23.302480 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:23.302572 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":305,"threads_started":1}
I20260927 18:48:23.304312 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:23.303486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:59743)
0927 18:48:23.304010 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:23.304105 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":268,"threads_started":1}
I20260927 18:48:23.802860 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:23.802085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:50941)
0927 18:48:23.802569 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:23.802678 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260927 18:48:24.304593 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:24.303832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:33989)
0927 18:48:24.304306 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:24.304410 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260927 18:48:24.306105 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:24.305336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:57887)
0927 18:48:24.305829 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:24.305934 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
I20260927 18:48:24.804633 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:24.803835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:34453)
0927 18:48:24.804302 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:24.804395 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260927 18:48:25.306479 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:25.305667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:39309)
0927 18:48:25.306221 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:25.306318 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":241,"threads_started":1}
I20260927 18:48:25.308032 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:25.307257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:46903)
0927 18:48:25.307746 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:25.307847 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":219,"threads_started":1}
I20260927 18:48:25.806783 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:25.805842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:50335)
0927 18:48:25.806452 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:25.806554 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":356,"threads_started":1}
I20260927 18:48:26.308621 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:26.307746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:49779)
0927 18:48:26.308203 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:26.308348 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260927 18:48:26.309952 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:26.309063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:51167)
0927 18:48:26.309683 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:26.309779 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":281,"threads_started":1}
I20260927 18:48:26.808766 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:26.807807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:34173)
0927 18:48:26.808303 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:26.808466 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
I20260927 18:48:27.310559 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.309759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:46593)
0927 18:48:27.310229 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.310372 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260927 18:48:27.311959 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.311113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:50043)
0927 18:48:27.311655 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.311773 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":301,"threads_started":1}
I20260927 18:48:27.810758 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.809980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.194:33797)
0927 18:48:27.810453 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.810569 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260927 18:48:28.286470 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14403
I20260927 18:48:28.312708 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.311749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.195:40361)
0927 18:48:28.312284 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:28.312443 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260927 18:48:28.313947 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.312906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44371 (local address 127.31.26.193:36795)
0927 18:48:28.313487 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:28.313671 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
I20260927 18:48:28.321136 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.288044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.31.26.193:34073)
0927 18:48:28.288765 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:28.288779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:28.288821 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:48:28.290578 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:28.290589 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:28.291066 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:28.291423 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:28.292947 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.294375 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.295699 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.296414 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.296562 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:28.313678 (+ 17116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:28.313688 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:28.313708 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:28.313763 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:28.317771 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:28.318589 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:28.318597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:28.318606 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:28.318787 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:28.319123 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:28.319127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:28.319129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:28.319690 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:28.319931 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:28.320314 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 18:48:28.320782 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":381,"threads_started":1}
I20260927 18:48:28.322628 14560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:28.408833 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14403
I20260927 18:48:28.426924 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14568
I20260927 18:48:28.445945 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.427842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.31.26.194:38785)
0927 18:48:28.428390 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:28.428420 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:28.428467 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:48:28.430121 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:28.430131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:28.430582 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:28.430960 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:28.432442 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.433845 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.435291 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.435986 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.436204 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:28.438933 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:28.438941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:28.438957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:28.439005 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:28.442302 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:28.443102 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:28.443110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:28.443117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:28.443306 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:28.443730 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:28.443736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:28.443739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:28.444308 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:28.444638 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:28.445194 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 18:48:28.445650 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":269,"threads_started":1}
I20260927 18:48:28.447429 14691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:28.534706 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14568
I20260927 18:48:28.555119 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14699
I20260927 18:48:28.576431 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.556309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.31.26.195:45089)
0927 18:48:28.556975 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:28.556990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:28.557032 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:48:28.558760 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:28.558770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:28.559249 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:28.559581 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:28.561047 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.562452 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.563982 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.564815 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.564991 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:28.568189 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:28.568199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:28.568218 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:28.568270 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:28.572198 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:28.573423 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:28.573431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:28.573439 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:28.573650 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:28.574207 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:28.574212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:28.574215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:28.574916 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:28.575185 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:28.575751 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 18:48:28.576130 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":323,"threads_started":1}
I20260927 18:48:28.578198 14822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:28.666399 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14699
2026-09-27T18:48:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25306 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:48:28 dist-test-slave-rfd5 krb5kdc[14885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:48:28 dist-test-slave-rfd5 krb5kdc[14885](info): set up 2 sockets
Sep 27 18:48:28 dist-test-slave-rfd5 krb5kdc[14885](info): commencing operation
krb5kdc: starting...
W20260927 18:48:29.785836 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:48:29 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:48:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:48:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:48:30.042021 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33699
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:30.393142 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:30.393482 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:30.393576 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:30.401060 14898 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:48:30.401263 14898 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:48:30.401304 14898 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:48:30.401350 14898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:30.401405 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:30.401476 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:30.401521 14898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:30.401573 14898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:30.412214 14898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:30.413600 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:30.414979 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:30.428442 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:30.428473 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:30.428525 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:30.429352 14898 server_base.cc:1034] running on GCE node
I20260927 18:48:30.430325 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:30.431934 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:30.433235 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790534910433140 us; error 65 us; skew 500 ppm
I20260927 18:48:30.433633 14898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:30 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:30.437605 14898 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:48:30.439193 14898 webserver.cc:460] Webserver started at http://127.31.26.254:46551/ using document root <none> and password file <none>
I20260927 18:48:30.439960 14898 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:30.440055 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:30.440351 14898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:30.443847 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "35bfc915f5854687b07999586e6f61cc"
format_stamp: "Formatted at 2026-09-27 18:48:30 on dist-test-slave-rfd5"
server_key: "9fe58c954edb63c4c1c267b0ce6c0014"
server_key_iv: "4596693853d36049e252c146fe3a5ca0"
server_key_version: "encryptionkey@0"
I20260927 18:48:30.444720 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "35bfc915f5854687b07999586e6f61cc"
format_stamp: "Formatted at 2026-09-27 18:48:30 on dist-test-slave-rfd5"
server_key: "9fe58c954edb63c4c1c267b0ce6c0014"
server_key_iv: "4596693853d36049e252c146fe3a5ca0"
server_key_version: "encryptionkey@0"
I20260927 18:48:30.451025 14898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 18:48:30.455468 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:30.457388 14898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:48:30.457656 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "35bfc915f5854687b07999586e6f61cc"
format_stamp: "Formatted at 2026-09-27 18:48:30 on dist-test-slave-rfd5"
server_key: "9fe58c954edb63c4c1c267b0ce6c0014"
server_key_iv: "4596693853d36049e252c146fe3a5ca0"
server_key_version: "encryptionkey@0"
I20260927 18:48:30.457954 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:30.492101 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:30.498006 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:30.498293 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:30.518289 14898 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:33699
I20260927 18:48:30.518306 14964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:33699 every 8 connection(s)
I20260927 18:48:30.519848 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:30.524132 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:30.527318 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14898
I20260927 18:48:30.527545 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:48:30.528247 31851 external_mini_cluster.cc:1428] Setting key b5cfa6bf64f149eeebe84d9ae4462a3e
I20260927 18:48:30.538241 14965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap starting.
I20260927 18:48:30.542717 14965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:30.544342 14965 log.cc:824] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:48:30 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:30.548262 14965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: No bootstrap required, opened a new log
I20260927 18:48:30.556128 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:30.530639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.31.26.254:33699)
0927 18:48:30.531536 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:30.531549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:30.531595 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:48:30.533503 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:30.533532 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:30.534043 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:30.534471 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:30.536535 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:30.538064 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:30.540241 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:30.541031 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:30.541297 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:30.547246 (+ 5949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:30.547255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:30.547273 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:30.547319 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:30.550930 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:30.552061 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:30.552069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:30.552077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:30.552292 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:30.552842 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:30.552846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:30.552848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:30.553570 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:30.553841 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:30.554381 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 18:48:30.554908 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":378,"threads_started":1}
I20260927 18:48:30.558672 14965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:30.559239 14965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:30.559378 14965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bfc915f5854687b07999586e6f61cc, State: Initialized, Role: FOLLOWER
I20260927 18:48:30.560065 14965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:30.560346 14965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:30.560494 14965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:30.560684 14965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:30.563278 14965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:30.563961 14965 leader_election.cc:302] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35bfc915f5854687b07999586e6f61cc; no voters:
I20260927 18:48:30.564623 14965 leader_election.cc:288] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:48:30.564791 14970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:30.565582 14970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 LEADER]: Becoming Leader. State: Replica: 35bfc915f5854687b07999586e6f61cc, State: Running, Role: LEADER
I20260927 18:48:30.566221 14970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:30.566937 14965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:30.569397 14971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35bfc915f5854687b07999586e6f61cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } } }
I20260927 18:48:30.569964 14971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:30.570624 14972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bfc915f5854687b07999586e6f61cc. Latest consensus state: current_term: 1 leader_uuid: "35bfc915f5854687b07999586e6f61cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } } }
I20260927 18:48:30.571077 14972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:30.577878 14978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:30.583854 14978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:30.594280 14978 catalog_manager.cc:1347] Generated new cluster ID: 30f23cb2ea904b19a473a287a837b00b
I20260927 18:48:30.594429 14978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:48:30.627179 14978 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:48:30.629426 14978 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:48:30.645061 14978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Generated new TSK 0
I20260927 18:48:30.646185 14978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:48:30.715529 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33699
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:31.081058 14991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:31.081480 14991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:31.081645 14991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:31.089325 14991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:31.089502 14991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:31.089684 14991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:48:31.100276 14991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:31.101888 14991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:31.103338 14991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:31.117606 14999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:31.117779 14997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:31.118129 14996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:31.118885 14991 server_base.cc:1034] running on GCE node
I20260927 18:48:31.119802 14991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:31.120951 14991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:31.122218 14991 hybrid_clock.cc:648] HybridClock initialized: now 1790534911122174 us; error 62 us; skew 500 ppm
I20260927 18:48:31.122576 14991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:31 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:31.126017 14991 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:48:31.127622 14991 webserver.cc:460] Webserver started at http://127.31.26.193:38731/ using document root <none> and password file <none>
I20260927 18:48:31.128392 14991 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:31.128551 14991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:31.128885 14991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:31.132213 14991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "a2bad98582144865898df17c690e1daa"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "41bd3af7537b11c8b0a477666ba780e4"
server_key_iv: "255da3496755db2fabeefad194582f57"
server_key_version: "encryptionkey@0"
I20260927 18:48:31.133059 14991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "a2bad98582144865898df17c690e1daa"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "41bd3af7537b11c8b0a477666ba780e4"
server_key_iv: "255da3496755db2fabeefad194582f57"
server_key_version: "encryptionkey@0"
I20260927 18:48:31.138772 14991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:48:31.143074 15006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:31.144917 14991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:48:31.145164 14991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "a2bad98582144865898df17c690e1daa"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "41bd3af7537b11c8b0a477666ba780e4"
server_key_iv: "255da3496755db2fabeefad194582f57"
server_key_version: "encryptionkey@0"
I20260927 18:48:31.145445 14991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:31.166579 14991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:31.171526 14991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:31.171815 14991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:31.172932 14991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:31.174582 14991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:31.174719 14991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:31.174871 14991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:31.174944 14991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:31.210543 14991 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:37459
I20260927 18:48:31.210596 15118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:37459 every 8 connection(s)
I20260927 18:48:31.212019 14991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:48:31.213913 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 14991
I20260927 18:48:31.214169 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:48:31.214870 31851 external_mini_cluster.cc:1428] Setting key 6b9710dd79513be29a8e5d4c418daace
Sep 27 18:48:31 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:31.239048 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:31.215291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:41919 (local address 127.31.26.254:33699)
0927 18:48:31.216379 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:31.216387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:31.216940 (+ 553us) server_negotiation.cc:407] Connection header received
0927 18:48:31.220109 (+ 3169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:31.220114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:31.220192 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:31.220311 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:31.222957 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.223752 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.226699 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.227152 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.227294 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:31.232238 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:31.232245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:31.232248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:31.232289 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:31.235185 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:31.236181 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:31.236187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:31.236189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:31.236336 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:31.236883 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:31.236888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:31.236890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:31.237356 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:31.237534 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:31.238383 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 18:48:31.238658 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":805,"threads_started":1}
I20260927 18:48:31.240180 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:31.215864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:41919)
0927 18:48:31.216767 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:31.216816 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:31.219672 (+ 2856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:31.220664 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:31.220702 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:31.221691 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:31.222683 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:31.222717 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.223953 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.223961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:31.226477 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:31.226504 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.227488 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.227497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:31.227888 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:31.228759 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:31.228802 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:31.231881 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:31.235414 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:31.235426 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:31.235454 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:31.236005 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:31.236470 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:31.236475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:31.236479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:31.236729 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:31.237510 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:31.237520 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:31.237941 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 18:48:31.238251 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 18:48:31.238901 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:48:31.242304 15119 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33699
I20260927 18:48:31.242717 15119 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:48:31.243599 15119 heartbeater.cc:511] Master 127.31.26.254:33699 requested a full tablet report, sending...
I20260927 18:48:31.246178 14930 ts_manager.cc:194] Registered new tserver with Master: a2bad98582144865898df17c690e1daa (127.31.26.193:37459)
I20260927 18:48:31.247766 14930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:41919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:48:31.273373 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33699
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:31.627373 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:31.627694 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:31.627794 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:31.635515 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:31.635659 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:31.635780 15125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:48:31.646320 15125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:31.647598 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:31.648931 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:31.663365 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:31.663455 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:31.663456 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:31.664669 15125 server_base.cc:1034] running on GCE node
I20260927 18:48:31.665468 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:31.666539 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:31.667855 15125 hybrid_clock.cc:648] HybridClock initialized: now 1790534911667799 us; error 40 us; skew 500 ppm
I20260927 18:48:31.668278 15125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:31 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:31.671954 15125 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:48:31.673611 15125 webserver.cc:460] Webserver started at http://127.31.26.194:44009/ using document root <none> and password file <none>
I20260927 18:48:31.674441 15125 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:31.674551 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:31.674854 15125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:31.678021 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "3ae0f152fd764ea28c2bbf684acff7f2"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "b0a7dad9512ca54d585fadb00b315a1a"
server_key_iv: "a360034b68437c61281a982922a4d853"
server_key_version: "encryptionkey@0"
I20260927 18:48:31.678750 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "3ae0f152fd764ea28c2bbf684acff7f2"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "b0a7dad9512ca54d585fadb00b315a1a"
server_key_iv: "a360034b68437c61281a982922a4d853"
server_key_version: "encryptionkey@0"
I20260927 18:48:31.684217 15125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:48:31.687965 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:31.689555 15125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:48:31.689761 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "3ae0f152fd764ea28c2bbf684acff7f2"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "b0a7dad9512ca54d585fadb00b315a1a"
server_key_iv: "a360034b68437c61281a982922a4d853"
server_key_version: "encryptionkey@0"
I20260927 18:48:31.689992 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:31.710788 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:31.716138 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:31.716526 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:31.717687 15125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:31.719417 15125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:31.719564 15125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:31.719741 15125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:31.719822 15125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:31.756238 15125 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:33545
I20260927 18:48:31.756287 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:33545 every 8 connection(s)
I20260927 18:48:31.757733 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:48:31.759382 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15125
I20260927 18:48:31.759598 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:48:31.760154 31851 external_mini_cluster.cc:1428] Setting key 9a8df0f37b068f677275879a211b7030
Sep 27 18:48:31 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:31.784839 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:31.760810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:60223 (local address 127.31.26.254:33699)
0927 18:48:31.761473 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:31.761479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:31.762822 (+ 1343us) server_negotiation.cc:407] Connection header received
0927 18:48:31.764985 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:31.764991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:31.765115 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:31.765224 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:31.767136 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.768013 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.770259 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.770818 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.770946 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:31.776173 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:31.776182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:31.776190 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:31.776245 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:31.780421 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:31.781749 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:31.781757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:31.781762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:31.781982 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:31.782580 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:31.782589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:31.782596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:31.783083 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:31.783285 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:31.784058 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 18:48:31.784370 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":415,"threads_started":1}
I20260927 18:48:31.785624 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:31.761527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:60223)
0927 18:48:31.762588 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:31.762633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:31.764653 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:31.765525 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:31.765535 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:31.766122 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:31.766911 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:31.766936 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.768206 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.768210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:31.770011 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:31.770034 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:31.771124 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:31.771134 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:31.771547 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:31.772458 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:31.772502 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:31.775785 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:31.780704 (+ 4919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:31.780713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:31.780731 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:31.781518 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:31.782156 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:31.782172 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:31.782174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:31.782418 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:31.783243 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:31.783251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:31.783623 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 18:48:31.783877 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 18:48:31.784335 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":468,"threads_started":1}
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
I20260927 18:48:31.787623 15253 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33699
I20260927 18:48:31.788031 15253 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:31.788908 15253 heartbeater.cc:511] Master 127.31.26.254:33699 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
I20260927 18:48:31.791016 14930 ts_manager.cc:194] Registered new tserver with Master: 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194:33545)
I20260927 18:48:31.792408 14930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:60223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:48:31.820273 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33699
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:32.173954 15259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:32.174338 15259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:32.174528 15259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:32.182220 15259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:32.182412 15259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:32.182598 15259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:48:32.192870 15259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:32.194423 15259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:32.195950 15259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:32.210338 15264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:32.210361 15265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:32.210214 15267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:32.211877 15259 server_base.cc:1034] running on GCE node
I20260927 18:48:32.212587 15259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:32.213694 15259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:32.214949 15259 hybrid_clock.cc:648] HybridClock initialized: now 1790534912214888 us; error 54 us; skew 500 ppm
I20260927 18:48:32.215330 15259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:32 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:32.218808 15259 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:48:32.220377 15259 webserver.cc:460] Webserver started at http://127.31.26.195:46419/ using document root <none> and password file <none>
I20260927 18:48:32.221236 15259 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:32.221393 15259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:32.221728 15259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:48:32.224892 15259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "b4f1a58809cc423eb039503122b37359"
format_stamp: "Formatted at 2026-09-27 18:48:32 on dist-test-slave-rfd5"
server_key: "65d426223623088ab20c4003c5f49d43"
server_key_iv: "7653c9f9ebf1fb09cfb036db9e35a149"
server_key_version: "encryptionkey@0"
I20260927 18:48:32.225922 15259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "b4f1a58809cc423eb039503122b37359"
format_stamp: "Formatted at 2026-09-27 18:48:32 on dist-test-slave-rfd5"
server_key: "65d426223623088ab20c4003c5f49d43"
server_key_iv: "7653c9f9ebf1fb09cfb036db9e35a149"
server_key_version: "encryptionkey@0"
I20260927 18:48:32.232012 15259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:48:32.236195 15274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:32.237928 15259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:48:32.238166 15259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "b4f1a58809cc423eb039503122b37359"
format_stamp: "Formatted at 2026-09-27 18:48:32 on dist-test-slave-rfd5"
server_key: "65d426223623088ab20c4003c5f49d43"
server_key_iv: "7653c9f9ebf1fb09cfb036db9e35a149"
server_key_version: "encryptionkey@0"
I20260927 18:48:32.238409 15259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:32.251557 15119 heartbeater.cc:503] Master 127.31.26.254:33699 was elected leader, sending a full tablet report...
I20260927 18:48:32.277853 15259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:32.282894 15259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:32.283218 15259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:32.284253 15259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:32.285825 15259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:32.285930 15259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:32.286080 15259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:32.286151 15259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:32.321357 15259 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35133
I20260927 18:48:32.321406 15386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35133 every 8 connection(s)
I20260927 18:48:32.322770 15259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:48:32.326164 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15259
I20260927 18:48:32.326385 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:48:32.327044 31851 external_mini_cluster.cc:1428] Setting key 4ffe0c081c0922a098266a29efdeb769
Sep 27 18:48:32 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:32.345710 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.325507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:60309 (local address 127.31.26.254:33699)
0927 18:48:32.326118 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.326125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.327220 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 18:48:32.329749 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.329755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.329847 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.329973 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:32.331835 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.332648 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.334328 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.334744 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.334850 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.338632 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:32.338639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:32.338642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:32.338684 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:32.341732 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:32.342916 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:32.342923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:32.342926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:32.343172 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:32.343758 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:32.343762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:32.343765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:32.344092 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:32.344232 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.345171 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.345395 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":308,"threads_started":1}
I20260927 18:48:32.346699 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.326133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:60309)
0927 18:48:32.327022 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:32.327079 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:32.329425 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:32.330246 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:32.330256 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:32.330994 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:32.331626 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.331646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.332810 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.332813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.334166 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.334180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.334954 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.334959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.335231 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.335962 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:32.335987 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:32.338340 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:32.341903 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:32.341914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:32.341934 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:32.342733 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:32.343319 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:32.343323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:32.343325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:32.343627 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:32.344246 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:32.344254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:32.344640 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 18:48:32.344907 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 18:48:32.345461 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":343,"threads_started":1}
I20260927 18:48:32.348394 15387 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33699
I20260927 18:48:32.348701 15387 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:32.349601 15387 heartbeater.cc:511] Master 127.31.26.254:33699 requested a full tablet report, sending...
I20260927 18:48:32.351315 14930 ts_manager.cc:194] Registered new tserver with Master: b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133)
I20260927 18:48:32.352149 14930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:60309
I20260927 18:48:32.356545 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:48:32.376129 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.362039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.31.26.254:33699)
0927 18:48:32.362506 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.362526 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.362713 (+ 187us) server_negotiation.cc:407] Connection header received
0927 18:48:32.362901 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.362913 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.363023 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.363127 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:32.364711 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.365667 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.367228 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.367739 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.367851 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.370511 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:32.370516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:32.370519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:32.370553 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:32.373169 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:32.373925 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:32.373929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:32.373931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:32.374053 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:32.374520 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:32.374527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:32.374533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:32.374896 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:32.375168 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.375553 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.375801 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 18:48:32.380790 14930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:48:32.388417 14930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:48:32.417263 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.403563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57730)
0927 18:48:32.407017 (+ 3454us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:32.407106 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:32.408120 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:32.409335 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:32.409342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:32.409406 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:32.410096 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.410112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.411658 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.411666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.414029 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.414043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.415896 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.415904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.416139 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.416215 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 18:48:32.416355 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:48:32.416578 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3316,"mutex_wait_us":81,"spinlock_wait_cycles":13952,"thread_start_us":4094,"threads_started":1}
I20260927 18:48:32.417992 15400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.403040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:33545 (local address 127.0.0.1:38974)
0927 18:48:32.407017 (+ 3977us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:32.408056 (+ 1039us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:32.408201 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:32.409009 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:32.409019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:32.409056 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:32.409595 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.409625 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.411795 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.411801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.414595 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.414613 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.416887 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.416897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.417236 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.417273 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:48:32.417356 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:48:32.417506 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3622,"mutex_wait_us":82,"thread_start_us":3422,"threads_started":1}
I20260927 18:48:32.418251 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.404358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.31.26.193:37459)
0927 18:48:32.405825 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.405841 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.407887 (+ 2046us) server_negotiation.cc:407] Connection header received
0927 18:48:32.408436 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.408447 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.408746 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.408945 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:32.410298 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.411450 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.414219 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.415639 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.415799 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.417530 (+ 1731us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.417647 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.417848 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":1061,"threads_started":1}
I20260927 18:48:32.418596 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.403631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.31.26.195:35133)
0927 18:48:32.406440 (+ 2809us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.406452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.408169 (+ 1717us) server_negotiation.cc:407] Connection header received
0927 18:48:32.408474 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.408483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.408756 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.408943 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:32.410454 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.412103 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.415020 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.416340 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.416475 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.416761 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.418016 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.418247 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2614,"thread_start_us":2447,"threads_started":1}
I20260927 18:48:32.418699 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.403042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35133 (local address 127.0.0.1:59580)
0927 18:48:32.408032 (+ 4990us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:32.408054 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:32.408272 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:32.409342 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:32.409345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:32.409410 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:32.410194 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.410215 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.412358 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.412367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.414834 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.414845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.416938 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.416945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.417383 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.417411 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:48:32.417473 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:48:32.417579 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4820,"mutex_wait_us":1329,"spinlock_wait_cycles":20736,"thread_start_us":3254,"threads_started":1}
I20260927 18:48:32.418831 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.403584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.31.26.194:33545)
0927 18:48:32.407469 (+ 3885us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.407480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.408372 (+ 892us) server_negotiation.cc:407] Connection header received
0927 18:48:32.408488 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.408495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.408748 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.408965 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:32.409846 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.411588 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.414812 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.416356 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.416508 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.416716 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.418304 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.418499 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3701,"thread_start_us":4029,"threads_started":1}
I20260927 18:48:32.422410 15054 tablet_service.cc:1467] Processing CreateTablet for tablet 529b238817594d5b9a267aff5f4a040e (DEFAULT_TABLE table=test-table [id=0115f02ea6d04b69aa5ea0239849d917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:32.422737 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 529b238817594d5b9a267aff5f4a040e (DEFAULT_TABLE table=test-table [id=0115f02ea6d04b69aa5ea0239849d917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:32.422735 15322 tablet_service.cc:1467] Processing CreateTablet for tablet 529b238817594d5b9a267aff5f4a040e (DEFAULT_TABLE table=test-table [id=0115f02ea6d04b69aa5ea0239849d917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:32.424124 15054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529b238817594d5b9a267aff5f4a040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:32.424456 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529b238817594d5b9a267aff5f4a040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:32.424453 15322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529b238817594d5b9a267aff5f4a040e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:32.436897 15406 tablet_bootstrap.cc:492] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Bootstrap starting.
I20260927 18:48:32.441016 15407 tablet_bootstrap.cc:492] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Bootstrap starting.
I20260927 18:48:32.441067 15406 tablet_bootstrap.cc:654] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:32.442518 15406 log.cc:824] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:32.443287 15408 tablet_bootstrap.cc:492] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Bootstrap starting.
I20260927 18:48:32.445187 15407 tablet_bootstrap.cc:654] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:32.446525 15407 log.cc:824] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:32.447309 15408 tablet_bootstrap.cc:654] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:32.447350 15406 tablet_bootstrap.cc:492] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: No bootstrap required, opened a new log
I20260927 18:48:32.447772 15406 ts_tablet_manager.cc:1397] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260927 18:48:32.448761 15408 log.cc:824] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:32.452189 15407 tablet_bootstrap.cc:492] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: No bootstrap required, opened a new log
I20260927 18:48:32.452499 15407 ts_tablet_manager.cc:1397] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 18:48:32.454084 15408 tablet_bootstrap.cc:492] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: No bootstrap required, opened a new log
I20260927 18:48:32.454385 15408 ts_tablet_manager.cc:1397] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260927 18:48:32.459653 15406 raft_consensus.cc:348] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.460194 15406 raft_consensus.cc:374] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:32.460352 15406 raft_consensus.cc:729] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bad98582144865898df17c690e1daa, State: Initialized, Role: FOLLOWER
I20260927 18:48:32.461151 15406 consensus_queue.cc:260] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.463570 15408 raft_consensus.cc:348] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.463758 15407 raft_consensus.cc:348] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.464195 15408 raft_consensus.cc:374] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:32.464301 15407 raft_consensus.cc:374] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:32.464474 15407 raft_consensus.cc:729] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f1a58809cc423eb039503122b37359, State: Initialized, Role: FOLLOWER
I20260927 18:48:32.464713 15408 raft_consensus.cc:729] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae0f152fd764ea28c2bbf684acff7f2, State: Initialized, Role: FOLLOWER
I20260927 18:48:32.464986 15406 ts_tablet_manager.cc:1428] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260927 18:48:32.465399 15407 consensus_queue.cc:260] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.465571 15408 consensus_queue.cc:260] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.466599 15387 heartbeater.cc:503] Master 127.31.26.254:33699 was elected leader, sending a full tablet report...
I20260927 18:48:32.467417 15407 ts_tablet_manager.cc:1428] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260927 18:48:32.468573 15253 heartbeater.cc:503] Master 127.31.26.254:33699 was elected leader, sending a full tablet report...
I20260927 18:48:32.469959 15408 ts_tablet_manager.cc:1428] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
W20260927 18:48:32.509955 15254 tablet.cc:2369] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:48:32.574633 15388 tablet.cc:2369] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:48:32.715106 15120 tablet.cc:2369] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:32.851925 15414 raft_consensus.cc:482] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:48:32.852380 15414 raft_consensus.cc:504] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.854485 15414 leader_election.cc:288] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133), a2bad98582144865898df17c690e1daa (127.31.26.193:37459)
I20260927 18:48:32.862370 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.854871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:47983 (local address 127.31.26.195:35133)
0927 18:48:32.855100 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.855109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.855630 (+ 521us) server_negotiation.cc:407] Connection header received
0927 18:48:32.855902 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.855908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.855971 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.856089 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:32.856678 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.857481 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.859701 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.861068 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.861303 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.861408 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.861954 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.862107 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:48:32.862370 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.855080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:57577 (local address 127.31.26.193:37459)
0927 18:48:32.855296 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.855305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.855987 (+ 682us) server_negotiation.cc:407] Connection header received
0927 18:48:32.856192 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.856195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.856259 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.856360 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:32.856959 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.857732 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.859885 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.861253 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.861417 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.861496 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.861954 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.862114 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:48:32.862609 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.854704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35133 (local address 127.31.26.194:47983)
0927 18:48:32.855506 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:32.855524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:32.855735 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:32.856105 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:32.856110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:32.856137 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:32.856514 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.856526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.857647 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.857653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.859513 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.859529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.861448 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.861456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.861683 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.861775 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 18:48:32.861879 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:48:32.862080 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":118,"thread_start_us":430,"threads_started":1}
I20260927 18:48:32.862780 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.855039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.31.26.194:57577)
0927 18:48:32.855866 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:32.855880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:32.856037 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:32.856377 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:32.856382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:32.856405 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:32.856802 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.856811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.857874 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.857881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.859680 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:32.859693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.861556 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.861562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:32.861769 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.861794 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:48:32.861879 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:48:32.862080 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":51,"thread_start_us":483,"threads_started":1}
I20260927 18:48:32.863925 15342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529b238817594d5b9a267aff5f4a040e" candidate_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f1a58809cc423eb039503122b37359" is_pre_election: true
I20260927 18:48:32.863957 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529b238817594d5b9a267aff5f4a040e" candidate_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bad98582144865898df17c690e1daa" is_pre_election: true
I20260927 18:48:32.864462 15342 raft_consensus.cc:2454] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae0f152fd764ea28c2bbf684acff7f2 in term 0.
I20260927 18:48:32.864521 15074 raft_consensus.cc:2454] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae0f152fd764ea28c2bbf684acff7f2 in term 0.
I20260927 18:48:32.865180 15141 leader_election.cc:302] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ae0f152fd764ea28c2bbf684acff7f2, b4f1a58809cc423eb039503122b37359; no voters:
I20260927 18:48:32.865602 15414 raft_consensus.cc:2781] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:48:32.865765 15414 raft_consensus.cc:482] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:48:32.865882 15414 raft_consensus.cc:3037] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:32.868450 15414 raft_consensus.cc:504] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.869144 15414 leader_election.cc:288] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [CANDIDATE]: Term 1 election: Requested vote from peers b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133), a2bad98582144865898df17c690e1daa (127.31.26.193:37459)
I20260927 18:48:32.869632 15342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529b238817594d5b9a267aff5f4a040e" candidate_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f1a58809cc423eb039503122b37359"
I20260927 18:48:32.869685 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529b238817594d5b9a267aff5f4a040e" candidate_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bad98582144865898df17c690e1daa"
I20260927 18:48:32.869858 15342 raft_consensus.cc:3037] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:32.869894 15074 raft_consensus.cc:3037] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:32.872087 15074 raft_consensus.cc:2454] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae0f152fd764ea28c2bbf684acff7f2 in term 1.
I20260927 18:48:32.872093 15342 raft_consensus.cc:2454] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae0f152fd764ea28c2bbf684acff7f2 in term 1.
I20260927 18:48:32.872591 15142 leader_election.cc:302] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ae0f152fd764ea28c2bbf684acff7f2, a2bad98582144865898df17c690e1daa; no voters:
I20260927 18:48:32.872929 15414 raft_consensus.cc:2781] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:32.873471 15414 raft_consensus.cc:686] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae0f152fd764ea28c2bbf684acff7f2, State: Running, Role: LEADER
I20260927 18:48:32.873972 15414 consensus_queue.cc:237] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } }
I20260927 18:48:32.878950 14929 catalog_manager.cc:5495] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } health_report { overall_health: UNKNOWN } } }
I20260927 18:48:32.988101 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:32.981933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.31.26.194:33545)
0927 18:48:32.982649 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:32.982654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:32.982673 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:48:32.982839 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:32.982843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:32.982931 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:32.983080 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:32.983483 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.984259 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.985822 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:32.986243 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:32.986345 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:32.986857 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:32.987433 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:48:32.987551 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:32.987685 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:48:32.987762 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":473,"threads_started":1}
I20260927 18:48:32.996942 15074 raft_consensus.cc:1261] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Refusing update from remote peer 3ae0f152fd764ea28c2bbf684acff7f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:32.996942 15342 raft_consensus.cc:1261] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Refusing update from remote peer 3ae0f152fd764ea28c2bbf684acff7f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:32.998145 15417 consensus_queue.cc:1035] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:32.998538 15414 consensus_queue.cc:1035] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:33.011548 15422 mvcc.cc:205] Tried to move back new op lower bound from 7334031003624550400 to 7334031003135791104. Current Snapshot: MvccSnapshot[applied={T|T < 7334031003624550400}]
I20260927 18:48:33.031543 14928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:33.031874 14928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:33.036079 14928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Sending DeleteTablet for 3 replicas of tablet 529b238817594d5b9a267aff5f4a040e
I20260927 18:48:33.037425 15322 tablet_service.cc:1514] Processing DeleteTablet for tablet 529b238817594d5b9a267aff5f4a040e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:33 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:59580
I20260927 18:48:33.037441 15054 tablet_service.cc:1514] Processing DeleteTablet for tablet 529b238817594d5b9a267aff5f4a040e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:33 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:57730
I20260927 18:48:33.037441 15188 tablet_service.cc:1514] Processing DeleteTablet for tablet 529b238817594d5b9a267aff5f4a040e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:33 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260927 18:48:33.040422 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14991
I20260927 18:48:33.051703 15435 tablet_replica.cc:330] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: stopping tablet replica
I20260927 18:48:33.052394 15435 raft_consensus.cc:2229] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:33.053148 15435 raft_consensus.cc:2258] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:33.054199 15434 tablet_replica.cc:330] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: stopping tablet replica
I20260927 18:48:33.054801 15434 raft_consensus.cc:2229] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:33.055488 15434 raft_consensus.cc:2258] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:33.059818 15434 ts_tablet_manager.cc:1905] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:33.059893 15436 tablet_replica.cc:330] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: stopping tablet replica
I20260927 18:48:33.060364 15436 raft_consensus.cc:2229] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:48:33.061205 15436 raft_consensus.cc:2258] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:33.062393 15435 ts_tablet_manager.cc:1905] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:33.065872 15436 ts_tablet_manager.cc:1905] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:48:33 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:48:33.068174 15434 ts_tablet_manager.cc:1918] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:33.068375 15434 log.cc:1196] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/529b238817594d5b9a267aff5f4a040e
I20260927 18:48:33.068949 15434 ts_tablet_manager.cc:1939] T 529b238817594d5b9a267aff5f4a040e P b4f1a58809cc423eb039503122b37359: Deleting consensus metadata
I20260927 18:48:33.071241 14914 catalog_manager.cc:4841] TS b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133): tablet 529b238817594d5b9a267aff5f4a040e (table test-table [id=0115f02ea6d04b69aa5ea0239849d917]) successfully deleted
I20260927 18:48:33.074707 15436 ts_tablet_manager.cc:1918] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:33.074934 15436 log.cc:1196] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/529b238817594d5b9a267aff5f4a040e
I20260927 18:48:33.075625 15436 ts_tablet_manager.cc:1939] T 529b238817594d5b9a267aff5f4a040e P 3ae0f152fd764ea28c2bbf684acff7f2: Deleting consensus metadata
I20260927 18:48:33.077800 14916 catalog_manager.cc:4841] TS 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194:33545): tablet 529b238817594d5b9a267aff5f4a040e (table test-table [id=0115f02ea6d04b69aa5ea0239849d917]) successfully deleted
I20260927 18:48:33.079149 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.041916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.31.26.193:37459)
0927 18:48:33.042186 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:33.042197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:33.054670 (+ 12473us) server_negotiation.cc:407] Connection header received
0927 18:48:33.055033 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:33.055040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:33.055142 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:33.055226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:33.057904 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:33.059071 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:33.061575 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:33.062345 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:33.062502 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:33.069586 (+ 7084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:33.069610 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:33.069631 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:33.069687 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:33.074242 (+ 4555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:33.075792 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:33.075804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:33.075808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:33.076151 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:33.076688 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:33.076694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:33.076698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:33.077290 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:33.077620 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:33.078392 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 18:48:33.078673 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 18:48:33.079355 15435 ts_tablet_manager.cc:1918] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:33.079581 15435 log.cc:1196] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/529b238817594d5b9a267aff5f4a040e
I20260927 18:48:33.079921 15114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:33.169919 15435 ts_tablet_manager.cc:1939] T 529b238817594d5b9a267aff5f4a040e P a2bad98582144865898df17c690e1daa: Deleting consensus metadata
I20260927 18:48:33.170462 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14991
W20260927 18:48:33.193662 14915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.26.193:37459 (error 108)
I20260927 18:48:33.194597 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 15125
I20260927 18:48:33.196364 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.195188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57746)
0927 18:48:33.195882 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.196093 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":781,"threads_started":1}
W20260927 18:48:33.196743 14915 catalog_manager.cc:4584] TS a2bad98582144865898df17c690e1daa (127.31.26.193:37459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 529b238817594d5b9a267aff5f4a040e: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Sep 27 18:48:33 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:48:33.215595 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.196033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.31.26.194:33545)
0927 18:48:33.196245 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:33.196252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:33.196276 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:48:33.196364 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:33.196371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:33.196479 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:33.196624 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:33.198644 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:33.199596 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:33.201422 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:33.201945 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:33.202128 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:33.208276 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:33.208297 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:33.208315 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:33.208365 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:33.212103 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:33.213027 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:33.213039 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:33.213041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:33.213272 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:33.213756 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:33.213761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:33.213763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:33.214168 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:33.214587 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:33.214907 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 18:48:33.215214 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:48:33.216406 15248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:33.250373 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.249887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57758)
0927 18:48:33.250107 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.250228 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 18:48:33.251840 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.251305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57768)
0927 18:48:33.251514 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.251626 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:48:33.312572 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.312061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57782)
0927 18:48:33.312322 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.312414 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 18:48:33.313894 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.313485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57788)
0927 18:48:33.313663 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.313751 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:48:33.333468 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 15125
I20260927 18:48:33.356761 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 15259
Sep 27 18:48:33 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:48:33.382138 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.357809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.31.26.195:35133)
0927 18:48:33.358023 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:33.358032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:33.358058 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:48:33.358133 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:33.358141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:33.358251 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:33.358394 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:33.359941 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:33.361422 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:33.362886 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:33.363754 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:33.363902 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:33.375105 (+ 11203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:33.375131 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:33.375149 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:33.375198 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:33.378559 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:33.379748 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:33.379757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:33.379759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:33.379919 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:33.380428 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:33.380433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:33.380435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:33.380885 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:33.381230 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:33.381634 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 18:48:33.381822 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:48:33.382997 15382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:33.394766 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.394289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57800)
0927 18:48:33.394514 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.394620 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:48:33.395885 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.395449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57816)
0927 18:48:33.395661 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.395738 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 18:48:33.474367 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 15259
I20260927 18:48:33.492767 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 14898
I20260927 18:48:33.494005 14960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:33.606933 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:33.606024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:57818)
0927 18:48:33.606443 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:33.606570 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:37459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 18:48:33.607437 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 14898
I20260927 18:48:53.626241 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33699
--webserver_interface=127.31.26.254
--webserver_port=46551
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:53.979045 15444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:53.979482 15444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:53.979652 15444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:53.987262 15444 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:48:53.987426 15444 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:48:53.987524 15444 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:48:53.987618 15444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:53.987722 15444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:53.987856 15444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:53.987947 15444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:53.988030 15444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:53.998780 15444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:54.000589 15444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:54.002550 15444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:54.028364 15450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:54.028297 15453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:54.028326 15451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:54.029124 15444 server_base.cc:1034] running on GCE node
I20260927 18:48:54.030341 15444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:54.032161 15444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:54.033447 15444 hybrid_clock.cc:648] HybridClock initialized: now 1790534934033350 us; error 53 us; skew 500 ppm
I20260927 18:48:54.034057 15444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:54 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534934, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:54.037923 15444 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:48:54.039666 15444 webserver.cc:460] Webserver started at http://127.31.26.254:46551/ using document root <none> and password file <none>
I20260927 18:48:54.040503 15444 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:54.040613 15444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:54.047616 15444 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:48:54.050992 15460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:54.052627 15444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 18:48:54.052851 15444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "35bfc915f5854687b07999586e6f61cc"
format_stamp: "Formatted at 2026-09-27 18:48:30 on dist-test-slave-rfd5"
server_key: "9fe58c954edb63c4c1c267b0ce6c0014"
server_key_iv: "4596693853d36049e252c146fe3a5ca0"
server_key_version: "encryptionkey@0"
I20260927 18:48:54.054061 15444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:54.089259 15444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:54.094530 15444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:54.094918 15444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:54.118333 15444 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:33699
I20260927 18:48:54.118402 15511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:33699 every 8 connection(s)
I20260927 18:48:54.120189 15444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:54.122403 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15444
I20260927 18:48:54.123271 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:37459
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:33699
--webserver_port=38731
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:48:54.130270 15512 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:48:54.135895 15512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap starting.
I20260927 18:48:54.157473 15512 log.cc:824] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:54.171115 15512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:48:54.171772 15512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap complete.
I20260927 18:48:54.181821 15512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:54.182787 15512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bfc915f5854687b07999586e6f61cc, State: Initialized, Role: FOLLOWER
I20260927 18:48:54.183604 15512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:54.183883 15512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:54.184051 15512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:54.184252 15512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:48:54.189071 15512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:54.189729 15512 leader_election.cc:302] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35bfc915f5854687b07999586e6f61cc; no voters:
I20260927 18:48:54.190433 15512 leader_election.cc:288] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:48:54.190582 15516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:48:54.191735 15516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 LEADER]: Becoming Leader. State: Replica: 35bfc915f5854687b07999586e6f61cc, State: Running, Role: LEADER
I20260927 18:48:54.192487 15516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:48:54.192958 15512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:54.195314 15518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bfc915f5854687b07999586e6f61cc. Latest consensus state: current_term: 2 leader_uuid: "35bfc915f5854687b07999586e6f61cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } } }
I20260927 18:48:54.195729 15518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:54.196317 15517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "35bfc915f5854687b07999586e6f61cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } } }
I20260927 18:48:54.196653 15517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:54.197414 15522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:54.220322 15522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:54.222448 15522 catalog_manager.cc:1259] Loaded cluster ID: 30f23cb2ea904b19a473a287a837b00b
I20260927 18:48:54.222711 15522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:48:54.228470 15522 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:48:54.230991 15522 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Loaded TSK: 0
I20260927 18:48:54.299710 15533 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:48:54.303016 15522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Generated new TSK 1
I20260927 18:48:54.303462 15522 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:48:54.544390 15514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:54.544719 15514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:54.544884 15514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:54.552958 15514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:54.553138 15514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:54.553345 15514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:48:54.564754 15514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:37459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=38731
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:54.566174 15514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:54.567610 15514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:54.582815 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:54.583039 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:54.582994 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:54.583802 15514 server_base.cc:1034] running on GCE node
I20260927 18:48:54.584578 15514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:54.585798 15514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:54.587049 15514 hybrid_clock.cc:648] HybridClock initialized: now 1790534934586999 us; error 53 us; skew 500 ppm
I20260927 18:48:54.587419 15514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:54 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534934, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:54.597903 15514 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:48:54.600311 15514 webserver.cc:460] Webserver started at http://127.31.26.193:38731/ using document root <none> and password file <none>
I20260927 18:48:54.601243 15514 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:54.601438 15514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:54.607982 15514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:48:54.611346 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:54.613158 15514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:48:54.613371 15514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "a2bad98582144865898df17c690e1daa"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "41bd3af7537b11c8b0a477666ba780e4"
server_key_iv: "255da3496755db2fabeefad194582f57"
server_key_version: "encryptionkey@0"
I20260927 18:48:54.614451 15514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:54.639516 15514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:54.644706 15514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:54.645112 15514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:54.646292 15514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:54.647797 15514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:54.647889 15514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:54.648027 15514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:54.648133 15514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:54.682626 15514 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:37459
I20260927 18:48:54.682677 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:37459 every 8 connection(s)
I20260927 18:48:54.684237 15514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:48:54.684937 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15514
I20260927 18:48:54.685784 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:33545
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:33699
--webserver_port=44009
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:48:54 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534934, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:54.713428 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:54.687822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:45913 (local address 127.31.26.254:33699)
0927 18:48:54.688525 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:54.688544 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:54.689222 (+ 678us) server_negotiation.cc:407] Connection header received
0927 18:48:54.691538 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:54.691560 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:54.692105 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:54.692559 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:54.694589 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:54.695912 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:54.698656 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:54.699340 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:54.699491 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:54.704493 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:54.704503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:54.704526 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:54.704576 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:54.707890 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:54.709255 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:54.709264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:54.709272 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:54.709472 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:54.710085 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:54.710089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:54.710091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:54.710615 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:54.710771 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:54.711907 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 18:48:54.712275 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":275,"threads_started":1}
I20260927 18:48:54.713817 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:54.687985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:45913)
0927 18:48:54.688976 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:54.689043 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:54.690887 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:54.692732 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:54.692741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:54.693510 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:54.694268 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:54.694300 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:54.696118 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:54.696127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:54.698447 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:54.698468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:54.699607 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:54.699616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:54.699961 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:54.700815 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:54.700853 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:54.704111 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:54.708126 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:54.708140 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:54.708163 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:54.708969 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:54.709614 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:54.709623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:54.709630 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:54.709948 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:54.710783 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:54.710794 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:54.711327 (+ 533us) client_negotiation.cc:769] Sending connection context
0927 18:48:54.711696 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 18:48:54.712419 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":392,"threads_started":1}
I20260927 18:48:54.716964 15662 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33699
I20260927 18:48:54.717342 15662 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:54.718170 15662 heartbeater.cc:511] Master 127.31.26.254:33699 requested a full tablet report, sending...
I20260927 18:48:54.720461 15477 ts_manager.cc:194] Registered new tserver with Master: a2bad98582144865898df17c690e1daa (127.31.26.193:37459)
I20260927 18:48:54.722115 15477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:45913
W20260927 18:48:55.043036 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:55.043352 15666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:55.043569 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:55.051076 15666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:55.051209 15666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:55.051398 15666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:48:55.061795 15666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:33545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=44009
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:55.063297 15666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:55.064711 15666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:55.079149 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:55.079166 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:55.079769 15674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:55.084575 15666 server_base.cc:1034] running on GCE node
I20260927 18:48:55.085444 15666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:55.086583 15666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:55.087874 15666 hybrid_clock.cc:648] HybridClock initialized: now 1790534935087797 us; error 68 us; skew 500 ppm
I20260927 18:48:55.088255 15666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:55 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534935, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:55.091715 15666 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:48:55.093667 15666 webserver.cc:460] Webserver started at http://127.31.26.194:44009/ using document root <none> and password file <none>
I20260927 18:48:55.094559 15666 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:55.094727 15666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:55.100795 15666 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 18:48:55.104017 15681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:55.105755 15666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:48:55.105976 15666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "3ae0f152fd764ea28c2bbf684acff7f2"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "b0a7dad9512ca54d585fadb00b315a1a"
server_key_iv: "a360034b68437c61281a982922a4d853"
server_key_version: "encryptionkey@0"
I20260927 18:48:55.107079 15666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:55.148500 15666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:55.153177 15666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:55.153487 15666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:55.154659 15666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:55.156100 15666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:55.156229 15666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:55.156375 15666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:55.156464 15666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:55.191637 15666 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:33545
I20260927 18:48:55.191740 15793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:33545 every 8 connection(s)
I20260927 18:48:55.193280 15666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:48:55.195025 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15666
I20260927 18:48:55.195688 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:35133
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:33699
--webserver_port=46419
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:48:55 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534935, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:55.217649 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.196228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:47789 (local address 127.31.26.254:33699)
0927 18:48:55.196449 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.196457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.197956 (+ 1499us) server_negotiation.cc:407] Connection header received
0927 18:48:55.201135 (+ 3179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.201143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.201278 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.201375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:55.203754 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.204483 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.206334 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.206777 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.206924 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.210878 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:55.210884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:55.210887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:55.210934 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:55.214108 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.215079 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.215083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.215086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.215229 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.215713 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.215724 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.215726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.216030 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:55.216198 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.216987 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.217290 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:48:55.218874 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.196827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:47789)
0927 18:48:55.197743 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.197797 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.200674 (+ 2877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.201701 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.201716 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.202589 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:55.203519 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.203549 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.204682 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.204686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.206155 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.206169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.207042 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.207049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.207382 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.208153 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:55.208179 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:55.210582 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:55.214322 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:55.214332 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:55.214353 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:55.214913 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:55.215353 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:55.215359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:55.215363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:55.215578 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:55.216188 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:55.216197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:55.216577 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.216884 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.217515 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":337,"threads_started":1}
I20260927 18:48:55.220652 15794 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33699
I20260927 18:48:55.221064 15794 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:55.222106 15794 heartbeater.cc:511] Master 127.31.26.254:33699 requested a full tablet report, sending...
I20260927 18:48:55.223905 15477 ts_manager.cc:194] Registered new tserver with Master: 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194:33545)
I20260927 18:48:55.224769 15477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:47789
W20260927 18:48:55.547264 15797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:55.547574 15797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:55.547746 15797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:55.555408 15797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:55.555544 15797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:55.555734 15797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:48:55.566399 15797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:35133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=46419
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:48:55.567768 15797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:55.569193 15797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:55.583397 15805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:55.583345 15803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:55.584097 15802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:55.584467 15797 server_base.cc:1034] running on GCE node
I20260927 18:48:55.585242 15797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:55.586282 15797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:55.587519 15797 hybrid_clock.cc:648] HybridClock initialized: now 1790534935587474 us; error 31 us; skew 500 ppm
I20260927 18:48:55.587894 15797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:48:55 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534935, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:55.591295 15797 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:48:55.592954 15797 webserver.cc:460] Webserver started at http://127.31.26.195:46419/ using document root <none> and password file <none>
I20260927 18:48:55.593726 15797 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:55.593900 15797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:55.599812 15797 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 18:48:55.602870 15812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:55.604496 15797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:48:55.604652 15797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "b4f1a58809cc423eb039503122b37359"
format_stamp: "Formatted at 2026-09-27 18:48:32 on dist-test-slave-rfd5"
server_key: "65d426223623088ab20c4003c5f49d43"
server_key_iv: "7653c9f9ebf1fb09cfb036db9e35a149"
server_key_version: "encryptionkey@0"
I20260927 18:48:55.605758 15797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:55.628139 15797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:55.632357 15797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:55.632603 15797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:55.633666 15797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:55.635067 15797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:55.635162 15797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:55.635349 15797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:55.635442 15797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:55.669946 15797 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35133
I20260927 18:48:55.670002 15924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35133 every 8 connection(s)
I20260927 18:48:55.671499 15797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:48:55.674042 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15797
Sep 27 18:48:55 dist-test-slave-rfd5 krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534935, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:48:55.692631 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.675357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.31.26.254:33699)
0927 18:48:55.677455 (+ 2098us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.677460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.677475 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:48:55.677528 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.677532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.677591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.677719 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:55.679334 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.680100 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.681529 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.682082 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.682216 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.685576 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:55.685584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:55.685590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:55.685630 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:55.689303 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.690403 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.690407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.690410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.690556 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.691032 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.691035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.691037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.691435 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:55.691692 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.691988 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.692237 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1909,"threads_started":1}
I20260927 18:48:55.694981 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.674321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:42731 (local address 127.31.26.254:33699)
0927 18:48:55.674554 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.674559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.675896 (+ 1337us) server_negotiation.cc:407] Connection header received
0927 18:48:55.677916 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.677921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.677999 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.678120 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:55.679647 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.680388 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.682029 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.682691 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.682840 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.687549 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:55.687556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:55.687561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:55.687616 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:55.691133 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.692176 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.692181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.692185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.692366 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.692972 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.692984 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.692986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.693332 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:55.693485 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.694354 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.694595 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:48:55.696214 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.674883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:42731)
0927 18:48:55.675716 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.675772 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.677645 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.678379 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.678387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.678910 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:55.679470 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.679488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.680558 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.680562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.681877 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.681889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.682956 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.682967 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.683308 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.684071 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:55.684105 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:55.687239 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:55.691291 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:55.691300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:55.691320 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:55.692015 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:55.692519 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:55.692523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:55.692525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:55.692830 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:55.693466 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:55.693475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:55.693975 (+ 500us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.694271 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.694901 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":343,"threads_started":1}
I20260927 18:48:55.697690 15925 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33699
I20260927 18:48:55.697992 15925 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:55.698691 15925 heartbeater.cc:511] Master 127.31.26.254:33699 requested a full tablet report, sending...
I20260927 18:48:55.700193 15477 ts_manager.cc:194] Registered new tserver with Master: b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133)
I20260927 18:48:55.701045 15477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:42731
I20260927 18:48:55.708168 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:48:55.716141 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.709273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.31.26.254:33699)
0927 18:48:55.709505 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.709514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.709944 (+ 430us) server_negotiation.cc:407] Connection header received
0927 18:48:55.710208 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.710214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.710391 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.710558 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:55.711014 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.711887 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.713941 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.714412 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.714568 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.715200 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:55.715425 (+ 225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 18:48:55.715874 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40844: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 18:48:55.716306 15664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40844: token expired
W20260927 18:48:55.716802 15391 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.26.254:33699)
I20260927 18:48:55.725548 15662 heartbeater.cc:503] Master 127.31.26.254:33699 was elected leader, sending a full tablet report...
I20260927 18:48:55.731865 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.717416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.31.26.254:33699)
0927 18:48:55.717606 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.717611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.717700 (+ 89us) server_negotiation.cc:407] Connection header received
0927 18:48:55.717935 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.717939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.718016 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.718124 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:55.719959 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.720730 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.722255 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.722682 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.722834 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.725968 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:55.725975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:55.725979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:55.726032 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:55.729075 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.729909 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.729913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.729915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.730052 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:55.730439 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:55.730442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:55.730444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:55.730785 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:55.730974 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.731267 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.731487 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:48:55.735818 15477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:48:55.743609 15477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:48:55.785755 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.760429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.31.26.194:33545)
0927 18:48:55.776750 (+ 16321us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.776759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.777360 (+ 601us) server_negotiation.cc:407] Connection header received
0927 18:48:55.777476 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.777484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.777734 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.777935 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:55.778703 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.780243 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.782544 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.784066 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.784212 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.784436 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.785169 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.785367 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16116,"thread_start_us":16937,"threads_started":1}
I20260927 18:48:55.785804 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.760149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:33545 (local address 127.0.0.1:59342)
0927 18:48:55.777050 (+ 16901us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.777068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.777218 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.777956 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.777961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.777992 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:55.778479 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.778493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.780411 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.780417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.782363 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.782376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.784600 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.784606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.784879 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.784947 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.785100 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.785391 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16776,"mutex_wait_us":97,"thread_start_us":13898,"threads_started":1}
I20260927 18:48:55.785892 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.757523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.31.26.193:37459)
0927 18:48:55.768881 (+ 11358us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.768894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.776544 (+ 7650us) server_negotiation.cc:407] Connection header received
0927 18:48:55.776745 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.776757 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.777126 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.777360 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:55.778417 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.780139 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.782952 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.784498 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.784640 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.784876 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.785354 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.785533 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11132,"thread_start_us":10979,"threads_started":1}
I20260927 18:48:55.786144 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.757321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35133 (local address 127.0.0.1:46224)
0927 18:48:55.773361 (+ 16040us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.773438 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.773751 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.778469 (+ 4718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.778476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.778505 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:55.779239 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.779252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.781399 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.781410 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.783440 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.783453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.785353 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.785359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.785532 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.785563 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.785625 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.785773 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15610,"thread_start_us":18705,"threads_started":1}
I20260927 18:48:55.786581 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.757367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:37459 (local address 127.0.0.1:36066)
0927 18:48:55.776391 (+ 19024us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.776420 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.776573 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.777646 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.777654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.777688 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:55.778218 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.778232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.780296 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.780304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.782769 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.782783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.784782 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.784789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.784987 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.785028 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.785194 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.786239 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18831,"mutex_wait_us":899,"spinlock_wait_cycles":27392,"thread_start_us":20212,"threads_started":1}
I20260927 18:48:55.787411 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.758147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.31.26.195:35133)
0927 18:48:55.777298 (+ 19151us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.777306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.777431 (+ 125us) server_negotiation.cc:407] Connection header received
0927 18:48:55.777535 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.777542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.778190 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.778479 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:55.779448 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.781238 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.783639 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.785075 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.786535 (+ 1460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.786785 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.786872 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.787065 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18974,"thread_start_us":19980,"threads_started":1}
I20260927 18:48:55.790516 15597 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07efd51c724d8682173e7827477efb (DEFAULT_TABLE table=test-table [id=d52b39e0482c48a38e5a5a30a4271259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:55.790524 15729 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07efd51c724d8682173e7827477efb (DEFAULT_TABLE table=test-table [id=d52b39e0482c48a38e5a5a30a4271259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:55.790851 15860 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07efd51c724d8682173e7827477efb (DEFAULT_TABLE table=test-table [id=d52b39e0482c48a38e5a5a30a4271259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:55.791778 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d07efd51c724d8682173e7827477efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:55.791849 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d07efd51c724d8682173e7827477efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:55.792047 15860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d07efd51c724d8682173e7827477efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:55.805617 15941 tablet_bootstrap.cc:492] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Bootstrap starting.
I20260927 18:48:55.809464 15941 tablet_bootstrap.cc:654] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:55.810611 15943 tablet_bootstrap.cc:492] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Bootstrap starting.
I20260927 18:48:55.811065 15942 tablet_bootstrap.cc:492] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Bootstrap starting.
I20260927 18:48:55.811306 15941 log.cc:824] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:55.814880 15941 tablet_bootstrap.cc:492] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: No bootstrap required, opened a new log
I20260927 18:48:55.815202 15941 ts_tablet_manager.cc:1397] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 18:48:55.815186 15942 tablet_bootstrap.cc:654] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:55.816316 15943 tablet_bootstrap.cc:654] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:55.816598 15942 log.cc:824] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:55.817623 15943 log.cc:824] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:55.820231 15942 tablet_bootstrap.cc:492] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: No bootstrap required, opened a new log
I20260927 18:48:55.820547 15942 ts_tablet_manager.cc:1397] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 18:48:55.820791 15943 tablet_bootstrap.cc:492] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: No bootstrap required, opened a new log
I20260927 18:48:55.821066 15943 ts_tablet_manager.cc:1397] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 18:48:55.824025 15941 raft_consensus.cc:348] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.824702 15941 raft_consensus.cc:374] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:55.824862 15941 raft_consensus.cc:729] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bad98582144865898df17c690e1daa, State: Initialized, Role: FOLLOWER
I20260927 18:48:55.825615 15941 consensus_queue.cc:260] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.827319 15941 ts_tablet_manager.cc:1428] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 18:48:55.831933 15942 raft_consensus.cc:348] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.832528 15942 raft_consensus.cc:374] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:55.832635 15942 raft_consensus.cc:729] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae0f152fd764ea28c2bbf684acff7f2, State: Initialized, Role: FOLLOWER
I20260927 18:48:55.833503 15942 consensus_queue.cc:260] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.834295 15943 raft_consensus.cc:348] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.834877 15794 heartbeater.cc:503] Master 127.31.26.254:33699 was elected leader, sending a full tablet report...
I20260927 18:48:55.834923 15943 raft_consensus.cc:374] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:55.835033 15943 raft_consensus.cc:729] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f1a58809cc423eb039503122b37359, State: Initialized, Role: FOLLOWER
I20260927 18:48:55.835723 15942 ts_tablet_manager.cc:1428] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 18:48:55.835753 15943 consensus_queue.cc:260] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.836974 15925 heartbeater.cc:503] Master 127.31.26.254:33699 was elected leader, sending a full tablet report...
I20260927 18:48:55.837786 15943 ts_tablet_manager.cc:1428] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
I20260927 18:48:55.922225 15947 raft_consensus.cc:482] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:48:55.922523 15947 raft_consensus.cc:504] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
W20260927 18:48:55.923197 15926 tablet.cc:2369] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:55.924422 15947 leader_election.cc:288] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133), 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194:33545)
I20260927 18:48:55.931869 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.925060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:51677 (local address 127.31.26.194:33545)
0927 18:48:55.925329 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.925338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.926033 (+ 695us) server_negotiation.cc:407] Connection header received
0927 18:48:55.926247 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.926251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.926310 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.926421 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:55.927014 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.927719 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.929993 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.930838 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.931043 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.931117 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.931434 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.931566 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 18:48:55.931869 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.924902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:45699 (local address 127.31.26.195:35133)
0927 18:48:55.925107 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.925117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.925699 (+ 582us) server_negotiation.cc:407] Connection header received
0927 18:48:55.925988 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.925993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.926051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.926152 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:55.926728 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.927465 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.929832 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.930811 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.930985 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.931078 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.931476 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.931603 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:48:55.932108 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.924936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:33545 (local address 127.31.26.193:51677)
0927 18:48:55.925911 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.925926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.926087 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.926443 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.926447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.926473 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:55.926858 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.926867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.927860 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.927868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.929750 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.929767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.931106 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.931115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.931256 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.931296 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.931374 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.931548 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":65,"thread_start_us":609,"threads_started":1}
I20260927 18:48:55.932132 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.924679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35133 (local address 127.31.26.193:45699)
0927 18:48:55.925573 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:55.925589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:55.925828 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:55.926171 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:55.926175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:55.926200 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:55.926582 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.926593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.927620 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.927626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.929643 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:55.929677 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.931057 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.931064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:55.931252 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.931306 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 18:48:55.931416 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:48:55.931559 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":47,"thread_start_us":556,"threads_started":1}
I20260927 18:48:55.933324 15749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d07efd51c724d8682173e7827477efb" candidate_uuid: "a2bad98582144865898df17c690e1daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" is_pre_election: true
I20260927 18:48:55.933326 15880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d07efd51c724d8682173e7827477efb" candidate_uuid: "a2bad98582144865898df17c690e1daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f1a58809cc423eb039503122b37359" is_pre_election: true
I20260927 18:48:55.933845 15880 raft_consensus.cc:2454] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bad98582144865898df17c690e1daa in term 0.
I20260927 18:48:55.933843 15749 raft_consensus.cc:2454] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bad98582144865898df17c690e1daa in term 0.
I20260927 18:48:55.934574 15550 leader_election.cc:302] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2bad98582144865898df17c690e1daa, b4f1a58809cc423eb039503122b37359; no voters:
I20260927 18:48:55.935077 15947 raft_consensus.cc:2781] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:48:55.935232 15947 raft_consensus.cc:482] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:48:55.935367 15947 raft_consensus.cc:3037] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:48:55.936946 15663 tablet.cc:2369] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:55.937966 15947 raft_consensus.cc:504] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
I20260927 18:48:55.938552 15947 leader_election.cc:288] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [CANDIDATE]: Term 1 election: Requested vote from peers b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133), 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194:33545)
I20260927 18:48:55.939263 15880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d07efd51c724d8682173e7827477efb" candidate_uuid: "a2bad98582144865898df17c690e1daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f1a58809cc423eb039503122b37359"
I20260927 18:48:55.939239 15749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d07efd51c724d8682173e7827477efb" candidate_uuid: "a2bad98582144865898df17c690e1daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae0f152fd764ea28c2bbf684acff7f2"
I20260927 18:48:55.939492 15749 raft_consensus.cc:3037] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:55.939492 15880 raft_consensus.cc:3037] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:55.942883 15749 raft_consensus.cc:2454] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2bad98582144865898df17c690e1daa in term 1.
I20260927 18:48:55.942883 15880 raft_consensus.cc:2454] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2bad98582144865898df17c690e1daa in term 1.
I20260927 18:48:55.943506 15552 leader_election.cc:302] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ae0f152fd764ea28c2bbf684acff7f2, a2bad98582144865898df17c690e1daa; no voters:
I20260927 18:48:55.943948 15947 raft_consensus.cc:2781] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:55.944446 15947 raft_consensus.cc:686] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 1 LEADER]: Becoming Leader. State: Replica: a2bad98582144865898df17c690e1daa, State: Running, Role: LEADER
I20260927 18:48:55.944962 15947 consensus_queue.cc:237] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } }
W20260927 18:48:55.945514 15795 tablet.cc:2369] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:55.950290 15477 catalog_manager.cc:5495] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa reported cstate change: term changed from 0 to 1, leader changed from <none> to a2bad98582144865898df17c690e1daa (127.31.26.193). New cstate: current_term: 1 leader_uuid: "a2bad98582144865898df17c690e1daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2bad98582144865898df17c690e1daa" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 37459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 } health_report { overall_health: UNKNOWN } } }
I20260927 18:48:55.981051 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:55.974895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.31.26.193:37459)
0927 18:48:55.975119 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:55.975124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:55.975144 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:48:55.975404 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:55.975408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:55.975504 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:55.975641 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:55.976064 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.976846 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.978499 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:55.978976 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:55.979146 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:55.979755 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:55.980341 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:48:55.980527 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:55.980654 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:48:55.980754 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:48:55.989704 15749 raft_consensus.cc:1261] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Refusing update from remote peer a2bad98582144865898df17c690e1daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:55.989703 15880 raft_consensus.cc:1261] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Refusing update from remote peer a2bad98582144865898df17c690e1daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:55.990782 15952 consensus_queue.cc:1035] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f1a58809cc423eb039503122b37359" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:55.991154 15947 consensus_queue.cc:1035] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae0f152fd764ea28c2bbf684acff7f2" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:56.022488 15477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:56.022779 15477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:56.026485 15477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Sending DeleteTablet for 3 replicas of tablet 7d07efd51c724d8682173e7827477efb
I20260927 18:48:56.027760 15860 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07efd51c724d8682173e7827477efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:56 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:46224
I20260927 18:48:56.027760 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07efd51c724d8682173e7827477efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:56 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:59342
I20260927 18:48:56.027827 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07efd51c724d8682173e7827477efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:56 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:36066
I20260927 18:48:56.032058 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 15514
I20260927 18:48:56.041381 15965 tablet_replica.cc:330] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: stopping tablet replica
I20260927 18:48:56.042075 15965 raft_consensus.cc:2229] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:56.042773 15965 raft_consensus.cc:2258] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:56.043313 15967 tablet_replica.cc:330] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: stopping tablet replica
I20260927 18:48:56.043677 15966 tablet_replica.cc:330] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: stopping tablet replica
I20260927 18:48:56.043798 15967 raft_consensus.cc:2229] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:48:56.044119 15966 raft_consensus.cc:2229] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:56.044669 15966 raft_consensus.cc:2258] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:56.044793 15967 raft_consensus.cc:2258] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:56.046868 15966 ts_tablet_manager.cc:1905] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:56.047262 15965 ts_tablet_manager.cc:1905] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:56.047914 15967 ts_tablet_manager.cc:1905] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:56.054337 15966 ts_tablet_manager.cc:1918] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:56.054557 15966 log.cc:1196] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/7d07efd51c724d8682173e7827477efb
I20260927 18:48:56.055226 15966 ts_tablet_manager.cc:1939] T 7d07efd51c724d8682173e7827477efb P b4f1a58809cc423eb039503122b37359: Deleting consensus metadata
I20260927 18:48:56.055590 15965 ts_tablet_manager.cc:1918] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:56.055828 15965 log.cc:1196] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/7d07efd51c724d8682173e7827477efb
I20260927 18:48:56.055891 15967 ts_tablet_manager.cc:1918] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:56.056070 15967 log.cc:1196] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/7d07efd51c724d8682173e7827477efb
I20260927 18:48:56.056581 15965 ts_tablet_manager.cc:1939] T 7d07efd51c724d8682173e7827477efb P 3ae0f152fd764ea28c2bbf684acff7f2: Deleting consensus metadata
I20260927 18:48:56.056669 15967 ts_tablet_manager.cc:1939] T 7d07efd51c724d8682173e7827477efb P a2bad98582144865898df17c690e1daa: Deleting consensus metadata
I20260927 18:48:56.057454 15461 catalog_manager.cc:4841] TS b4f1a58809cc423eb039503122b37359 (127.31.26.195:35133): tablet 7d07efd51c724d8682173e7827477efb (table test-table [id=d52b39e0482c48a38e5a5a30a4271259]) successfully deleted
I20260927 18:48:56.058580 15462 catalog_manager.cc:4841] TS a2bad98582144865898df17c690e1daa (127.31.26.193:37459): tablet 7d07efd51c724d8682173e7827477efb (table test-table [id=d52b39e0482c48a38e5a5a30a4271259]) successfully deleted
I20260927 18:48:56.058681 15463 catalog_manager.cc:4841] TS 3ae0f152fd764ea28c2bbf684acff7f2 (127.31.26.194:33545): tablet 7d07efd51c724d8682173e7827477efb (table test-table [id=d52b39e0482c48a38e5a5a30a4271259]) successfully deleted
I20260927 18:48:56.061297 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:56.033742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.31.26.193:37459)
0927 18:48:56.033995 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:56.034005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:56.034031 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:48:56.034121 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:56.034129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:56.034239 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:56.034396 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:56.037219 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:56.038900 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:56.041837 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:56.042925 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:56.043096 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:56.052358 (+ 9262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:56.052374 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:56.052396 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:56.052455 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:56.056944 (+ 4489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:56.058128 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:56.058138 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:56.058142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:56.058355 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:56.059422 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:56.059427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:56.059431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:56.059948 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:56.060273 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:56.060625 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 18:48:56.060901 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:48:56.062145 15657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:56.163962 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 15514
I20260927 18:48:56.182968 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 15666
I20260927 18:48:56.202692 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:56.183774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.31.26.194:33545)
0927 18:48:56.183948 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:56.183952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:56.183975 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:48:56.184143 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:56.184147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:56.184242 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:56.184331 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:56.185889 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:56.186715 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:56.188086 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:56.188484 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:56.188583 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:56.195889 (+ 7306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:56.195904 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:56.195920 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:56.195974 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:56.199436 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:56.200325 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:56.200337 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:56.200346 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:56.200491 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:56.200938 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:56.200945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:56.200953 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:56.201452 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:56.201764 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:56.202083 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 18:48:56.202326 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:48:56.203424 15789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:56.296455 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 15666
I20260927 18:48:56.314476 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 15797
I20260927 18:48:56.328828 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:56.315274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.31.26.195:35133)
0927 18:48:56.315443 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:56.315447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:56.315468 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:48:56.315605 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:56.315609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:56.315671 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:56.315771 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:56.317074 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:56.317883 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:56.319240 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:56.319623 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:56.319722 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:56.323016 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:56.323032 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:56.323049 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:56.323088 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:56.326112 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:56.326953 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:56.326961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:56.326963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:56.327120 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:56.327481 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:56.327484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:56.327486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:56.327832 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:56.328062 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:56.328388 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:48:56.328573 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:48:56.329715 15920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:56.420648 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 15797
I20260927 18:48:56.438068 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 15444
I20260927 18:48:56.439216 15507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:48:56.554550 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 15444
I20260927 18:49:16.577155 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33699
--webserver_interface=127.31.26.254
--webserver_port=46551
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:16.926154 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:16.926555 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:16.926723 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:16.934513 15972 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:49:16.934682 15972 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:49:16.934779 15972 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:49:16.934877 15972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:16.934989 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:16.935099 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:16.935200 15972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:16.935288 15972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:16.946094 15972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=46551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:16.947548 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:16.949019 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:16.961786 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:16.961941 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:16.961915 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:16.963651 15972 server_base.cc:1034] running on GCE node
I20260927 18:49:16.964529 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:16.966210 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:16.967507 15972 hybrid_clock.cc:648] HybridClock initialized: now 1790534956967411 us; error 67 us; skew 500 ppm
I20260927 18:49:16.967945 15972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:16 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:16.978531 15972 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:49:16.980423 15972 webserver.cc:460] Webserver started at http://127.31.26.254:46551/ using document root <none> and password file <none>
I20260927 18:49:16.981364 15972 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:16.981513 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:16.987643 15972 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 18:49:16.990957 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:16.992573 15972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:49:16.992739 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "35bfc915f5854687b07999586e6f61cc"
format_stamp: "Formatted at 2026-09-27 18:48:30 on dist-test-slave-rfd5"
server_key: "9fe58c954edb63c4c1c267b0ce6c0014"
server_key_iv: "4596693853d36049e252c146fe3a5ca0"
server_key_version: "encryptionkey@0"
I20260927 18:49:16.993839 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:17.010589 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:17.015516 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:17.015789 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:17.036322 15972 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:33699
I20260927 18:49:17.036357 16038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:33699 every 8 connection(s)
I20260927 18:49:17.038074 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:17.042557 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 15972
I20260927 18:49:17.043402 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:37459
--local_ip_for_outbound_sockets=127.31.26.193
--tserver_master_addrs=127.31.26.254:33699
--webserver_port=38731
--webserver_interface=127.31.26.193
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:17.045658 16039 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:49:17.049172 16039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap starting.
I20260927 18:49:17.070529 16039 log.cc:824] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:17.087723 16039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:49:17.088337 16039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Bootstrap complete.
I20260927 18:49:17.099833 16039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:49:17.100857 16039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bfc915f5854687b07999586e6f61cc, State: Initialized, Role: FOLLOWER
I20260927 18:49:17.101588 16039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:49:17.101857 16039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:17.101990 16039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:17.102185 16039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:49:17.106351 16039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:49:17.106901 16039 leader_election.cc:302] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35bfc915f5854687b07999586e6f61cc; no voters:
I20260927 18:49:17.107623 16039 leader_election.cc:288] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:49:17.107795 16043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:49:17.110234 16039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:17.114985 16043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [term 3 LEADER]: Becoming Leader. State: Replica: 35bfc915f5854687b07999586e6f61cc, State: Running, Role: LEADER
I20260927 18:49:17.115782 16043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } }
I20260927 18:49:17.120092 16044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "35bfc915f5854687b07999586e6f61cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } } }
I20260927 18:49:17.120582 16044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:17.121311 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bfc915f5854687b07999586e6f61cc. Latest consensus state: current_term: 3 leader_uuid: "35bfc915f5854687b07999586e6f61cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35bfc915f5854687b07999586e6f61cc" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33699 } } }
I20260927 18:49:17.121708 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:17.121639 16053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:17.134297 16053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:17.136330 16053 catalog_manager.cc:1259] Loaded cluster ID: 30f23cb2ea904b19a473a287a837b00b
I20260927 18:49:17.136554 16053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:17.141041 16053 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:17.142881 16053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Loaded TSK: 0
I20260927 18:49:17.143163 16053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35bfc915f5854687b07999586e6f61cc: Loaded TSK: 1
F20260927 18:49:17.144331 16053 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc4208599ad google::LogMessage::Fail() at ??:0
@ 0x7fc42085c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc4208594de google::LogMessage::Flush() at ??:0
@ 0x7fc42085aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260927 18:49:17.152586 16060 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fc4245bd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc4245ac73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc4362881c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc436287783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc4363132a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fc4363130c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc43635d5e9 std::function<>::operator()() at ??:0
@ 0x7fc43628bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc436289c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc436311610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc436311439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc4234a5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc4234a891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc4234a8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc423483eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc42675d6db start_thread at ??:0
@ 0x7fc41f6f361f clone at ??:0
W20260927 18:49:17.448091 16041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:17.448453 16041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:17.448638 16041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:17.456331 16041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:17.456514 16041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:17.456754 16041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:17.467329 16041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:37459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=38731
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:17.468755 16041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:17.470290 16041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:17.484568 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:17.484501 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:17.484545 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:17.485663 16041 server_base.cc:1034] running on GCE node
I20260927 18:49:17.486431 16041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:17.487500 16041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:17.488759 16041 hybrid_clock.cc:648] HybridClock initialized: now 1790534957488710 us; error 45 us; skew 500 ppm
I20260927 18:49:17.489200 16041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:17 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:17.492803 16041 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:49:17.495191 16041 webserver.cc:460] Webserver started at http://127.31.26.193:38731/ using document root <none> and password file <none>
I20260927 18:49:17.496003 16041 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:17.496176 16041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:17.502244 16041 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:49:17.505478 16096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:17.506994 16041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:49:17.507205 16041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "a2bad98582144865898df17c690e1daa"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "41bd3af7537b11c8b0a477666ba780e4"
server_key_iv: "255da3496755db2fabeefad194582f57"
server_key_version: "encryptionkey@0"
I20260927 18:49:17.508291 16041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:17.521978 16041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:17.526423 16041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:17.526743 16041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:17.527802 16041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:17.529311 16041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:17.529482 16041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:17.529628 16041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:17.529728 16041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:17.565449 16041 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:37459
I20260927 18:49:17.565515 16208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:37459 every 8 connection(s)
I20260927 18:49:17.567067 16041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:17.569429 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16041
I20260927 18:49:17.570103 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:33545
--local_ip_for_outbound_sockets=127.31.26.194
--tserver_master_addrs=127.31.26.254:33699
--webserver_port=44009
--webserver_interface=127.31.26.194
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:17.573025 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:17.570743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:42415)
0927 18:49:17.571670 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:17.571980 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":402,"threads_started":1}
W20260927 18:49:17.573951 16209 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:33699 (0 consecutive failures): Network error: Failed to ping master at 127.31.26.254:33699: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
I20260927 18:49:17.575019 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:17.574647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:47561)
0927 18:49:17.574817 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:17.574895 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 18:49:17.575964 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:17.575588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:52129)
0927 18:49:17.575741 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:17.575834 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:49:17.576206 16209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:49:17.924288 16212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:17.924666 16212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:17.924839 16212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:17.932451 16212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:17.932636 16212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:17.932823 16212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:17.943786 16212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:33545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=44009
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:17.945298 16212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:17.946729 16212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:17.962258 16217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:17.962381 16218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:17.962584 16220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:17.963444 16212 server_base.cc:1034] running on GCE node
I20260927 18:49:17.964109 16212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:17.965193 16212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:17.966466 16212 hybrid_clock.cc:648] HybridClock initialized: now 1790534957966410 us; error 64 us; skew 500 ppm
I20260927 18:49:17.966838 16212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:17 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:17.970306 16212 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:49:17.972020 16212 webserver.cc:460] Webserver started at http://127.31.26.194:44009/ using document root <none> and password file <none>
I20260927 18:49:17.972792 16212 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:17.972962 16212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:17.978868 16212 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:49:17.982089 16227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:17.983654 16212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:17.983860 16212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "3ae0f152fd764ea28c2bbf684acff7f2"
format_stamp: "Formatted at 2026-09-27 18:48:31 on dist-test-slave-rfd5"
server_key: "b0a7dad9512ca54d585fadb00b315a1a"
server_key_iv: "a360034b68437c61281a982922a4d853"
server_key_version: "encryptionkey@0"
I20260927 18:49:17.984938 16212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:18.021342 16212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:18.026269 16212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:18.026608 16212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:18.027714 16212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:18.029280 16212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:18.029414 16212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:18.029558 16212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:18.029649 16212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:18.064986 16212 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:33545
I20260927 18:49:18.065104 16339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:33545 every 8 connection(s)
I20260927 18:49:18.066586 16212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:18.067195 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16212
I20260927 18:49:18.068019 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:35133
--local_ip_for_outbound_sockets=127.31.26.195
--tserver_master_addrs=127.31.26.254:33699
--webserver_port=46419
--webserver_interface=127.31.26.195
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:18.072595 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.070474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:46991)
0927 18:49:18.071354 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.071599 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
W20260927 18:49:18.073437 16340 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:33699 (0 consecutive failures): Network error: Failed to ping master at 127.31.26.254:33699: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
I20260927 18:49:18.074534 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.074089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:54845)
0927 18:49:18.074304 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.074396 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 18:49:18.075707 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.075267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:56275)
0927 18:49:18.075470 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.075574 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 18:49:18.076056 16340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:49:18.421720 16343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:18.422039 16343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:18.422211 16343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:18.429669 16343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:18.429800 16343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:18.430001 16343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:18.440434 16343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:35133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=46419
--tserver_master_addrs=127.31.26.254:33699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:18.441890 16343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:18.443358 16343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:18.457722 16348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:18.457638 16351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:18.457695 16349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:18.458735 16343 server_base.cc:1034] running on GCE node
I20260927 18:49:18.459493 16343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:18.460494 16343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:18.461696 16343 hybrid_clock.cc:648] HybridClock initialized: now 1790534958461622 us; error 67 us; skew 500 ppm
I20260927 18:49:18.462013 16343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:18 dist-test-slave-rfd5 krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:18.465485 16343 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:49:18.467134 16343 webserver.cc:460] Webserver started at http://127.31.26.195:46419/ using document root <none> and password file <none>
I20260927 18:49:18.467900 16343 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:18.468039 16343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:18.474305 16343 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:49:18.477541 16358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:18.479223 16343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:49:18.479454 16343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "b4f1a58809cc423eb039503122b37359"
format_stamp: "Formatted at 2026-09-27 18:48:32 on dist-test-slave-rfd5"
server_key: "65d426223623088ab20c4003c5f49d43"
server_key_iv: "7653c9f9ebf1fb09cfb036db9e35a149"
server_key_version: "encryptionkey@0"
I20260927 18:49:18.480580 16343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:18.494205 16343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:18.498960 16343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:18.499251 16343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:18.500268 16343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:18.501726 16343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:18.501823 16343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:18.501956 16343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:18.502068 16343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:18.537590 16343 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35133
I20260927 18:49:18.537634 16470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35133 every 8 connection(s)
I20260927 18:49:18.539176 16343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:18.544569 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.542476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:41055)
0927 18:49:18.543348 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.543591 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":377,"threads_started":1}
W20260927 18:49:18.545475 16471 heartbeater.cc:644] Failed to heartbeat to 127.31.26.254:33699 (0 consecutive failures): Network error: Failed to ping master at 127.31.26.254:33699: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
I20260927 18:49:18.545687 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16343
I20260927 18:49:18.546640 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.546156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:48131)
0927 18:49:18.546366 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.546472 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 18:49:18.547843 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.547446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:56989)
0927 18:49:18.547610 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.547710 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:49:18.548091 16471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:49:18.577879 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:18.576913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:54305)
0927 18:49:18.577499 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:18.577630 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":310,"threads_started":1}
I20260927 18:49:19.077658 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.076692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:36077)
0927 18:49:19.077299 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.077396 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":291,"threads_started":1}
I20260927 18:49:19.549860 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.548825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:59023)
0927 18:49:19.549510 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.549604 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":366,"threads_started":1}
I20260927 18:49:19.579885 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.579081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:55613)
0927 18:49:19.579606 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.579697 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
I20260927 18:49:20.079519 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:20.078789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:34439)
0927 18:49:20.079247 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:20.079349 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":230,"threads_started":1}
I20260927 18:49:20.551764 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:20.550916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:59261)
0927 18:49:20.551488 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:20.551580 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
I20260927 18:49:20.581974 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:20.581031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:43045)
0927 18:49:20.581578 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:20.581671 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":326,"threads_started":1}
I20260927 18:49:21.081493 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:21.080629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:50351)
0927 18:49:21.081208 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:21.081308 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260927 18:49:21.553751 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:21.552880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:55303)
0927 18:49:21.553480 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:21.553572 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":368,"threads_started":1}
I20260927 18:49:21.583963 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:21.583146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:38719)
0927 18:49:21.583659 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:21.583753 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260927 18:49:22.083279 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:22.082535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:33047)
0927 18:49:22.083019 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:22.083107 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":234,"threads_started":1}
I20260927 18:49:22.555722 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:22.554869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:41721)
0927 18:49:22.555415 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:22.555514 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":281,"threads_started":1}
I20260927 18:49:22.586019 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:22.585207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:42009)
0927 18:49:22.585744 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:22.585836 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":267,"threads_started":1}
I20260927 18:49:23.085333 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:23.084460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:41725)
0927 18:49:23.084997 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:23.085144 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":258,"threads_started":1}
I20260927 18:49:23.557894 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:23.556967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:58205)
0927 18:49:23.557550 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:23.557642 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":316,"threads_started":1}
I20260927 18:49:23.587971 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:23.587163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:59967)
0927 18:49:23.587687 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:23.587794 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
I20260927 18:49:24.087199 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.086455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:56155)
0927 18:49:24.086927 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.087032 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260927 18:49:24.560202 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.559037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:60411)
0927 18:49:24.559923 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.560026 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":552,"threads_started":1}
I20260927 18:49:24.590478 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.589235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:54027)
0927 18:49:24.590105 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.590228 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":275,"threads_started":1}
I20260927 18:49:25.089524 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.088432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:58977)
0927 18:49:25.089142 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.089249 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":300,"threads_started":1}
I20260927 18:49:25.562288 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.561430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:42995)
0927 18:49:25.561943 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.562065 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260927 18:49:25.592469 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.591674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:42585)
0927 18:49:25.592181 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.592283 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":247,"threads_started":1}
I20260927 18:49:26.091355 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.090579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:36561)
0927 18:49:26.091090 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.091182 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":279,"threads_started":1}
I20260927 18:49:26.564240 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.563481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:41405)
0927 18:49:26.563958 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.564062 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":243,"threads_started":1}
I20260927 18:49:26.594496 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.593590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:45757)
0927 18:49:26.594221 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.594313 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":328,"threads_started":1}
I20260927 18:49:27.093390 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.092408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:51209)
0927 18:49:27.093010 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.093200 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":348,"threads_started":1}
I20260927 18:49:27.566331 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.565433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:35707)
0927 18:49:27.566003 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.566156 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":256,"threads_started":1}
I20260927 18:49:27.596563 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.595721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:50047)
0927 18:49:27.596234 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.596324 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":293,"threads_started":1}
I20260927 18:49:28.095299 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:28.094416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:58137)
0927 18:49:28.095043 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:28.095128 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":389,"threads_started":1}
I20260927 18:49:28.568486 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:28.567589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:40145)
0927 18:49:28.568160 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:28.568275 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
I20260927 18:49:28.598544 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:28.597772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:54115)
0927 18:49:28.598263 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:28.598363 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260927 18:49:29.097477 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:29.096524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:59271)
0927 18:49:29.097076 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:29.097269 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
I20260927 18:49:29.570446 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:29.569688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:43093)
0927 18:49:29.570167 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:29.570260 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":218,"threads_started":1}
I20260927 18:49:29.600601 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:29.599628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:51343)
0927 18:49:29.600359 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:29.600445 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":517,"threads_started":1}
I20260927 18:49:30.099313 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:30.098543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:48085)
0927 18:49:30.099047 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:30.099140 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260927 18:49:30.572178 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:30.571410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:44891)
0927 18:49:30.571907 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:30.572002 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260927 18:49:30.602694 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:30.601769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:57821)
0927 18:49:30.602376 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:30.602498 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":341,"threads_started":1}
I20260927 18:49:31.101393 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:31.100432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:52465)
0927 18:49:31.100984 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:31.101152 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":260,"threads_started":1}
I20260927 18:49:31.574292 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:31.573449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:36577)
0927 18:49:31.573962 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:31.574122 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":253,"threads_started":1}
I20260927 18:49:31.604537 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:31.603690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:39037)
0927 18:49:31.604238 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:31.604336 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260927 18:49:32.103346 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:32.102418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:32903)
0927 18:49:32.103005 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:32.103123 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":254,"threads_started":1}
I20260927 18:49:32.576179 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:32.575386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:52561)
0927 18:49:32.575878 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:32.575984 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260927 18:49:32.606539 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:32.605818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:34909)
0927 18:49:32.606262 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:32.606352 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":239,"threads_started":1}
I20260927 18:49:33.105471 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.104577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.194:48721)
0927 18:49:33.105192 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:33.105285 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":309,"threads_started":1}
I20260927 18:49:33.547382 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16041
I20260927 18:49:33.570052 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.549831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.31.26.193:37459)
0927 18:49:33.550602 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:33.550625 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:33.550677 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:49:33.552430 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:33.552442 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:33.552988 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:33.553387 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:33.554996 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:33.556336 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:33.557958 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:33.558708 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:33.558875 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:33.562113 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:33.562122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:33.562142 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:33.562189 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:33.565659 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:33.566550 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:33.566558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:33.566565 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:33.566850 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:33.567446 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:33.567451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:33.567455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:33.568273 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:33.568637 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:33.569111 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 18:49:33.569602 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":641,"threads_started":1}
I20260927 18:49:33.571794 16204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:33.578485 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.577620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.195:59103)
0927 18:49:33.578186 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:33.578303 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":332,"threads_started":1}
I20260927 18:49:33.659133 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.657985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33699 (local address 127.31.26.193:44767)
0927 18:49:33.658563 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:33.658794 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.254:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":256,"threads_started":1}
I20260927 18:49:33.659150 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16041
I20260927 18:49:33.676766 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16212
I20260927 18:49:33.695886 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.677865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.31.26.194:33545)
0927 18:49:33.678418 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:33.678433 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:33.678473 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:49:33.680118 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:33.680129 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:33.680592 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:33.680974 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:33.682586 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:33.683858 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:33.685472 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:33.686183 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:33.686400 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:33.689229 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:33.689240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:33.689258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:33.689356 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:33.692251 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:33.692998 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:33.693014 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:33.693022 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:33.693273 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:33.693673 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:33.693678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:33.693682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:33.694252 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:33.694608 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:33.695078 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 18:49:33.695568 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
I20260927 18:49:33.697508 16335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:33.783306 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16212
I20260927 18:49:33.801808 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16343
I20260927 18:49:33.824354 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.803157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.31.26.195:35133)
0927 18:49:33.803859 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:33.803875 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:33.803926 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:49:33.805877 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:33.805889 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:33.806429 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:33.806862 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:33.808833 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:33.810290 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:33.811786 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:33.812568 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:33.812807 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:33.816257 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:33.816268 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:33.816294 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:33.816354 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:33.820283 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:33.821232 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:33.821240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:33.821249 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:33.821528 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:33.821981 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:33.821990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:33.821993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:33.822658 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:33.823002 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:33.823552 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 18:49:33.824040 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
I20260927 18:49:33.825995 16466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:33.914270 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16343
2026-09-27T18:49:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65248 ms)
[----------] 30 tests from SecurityITest (304979 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:34 dist-test-slave-rfd5 krb5kdc[16529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:34 dist-test-slave-rfd5 krb5kdc[16529](info): set up 2 sockets
Sep 27 18:49:34 dist-test-slave-rfd5 krb5kdc[16529](info): commencing operation
krb5kdc: starting...
W20260927 18:49:35.037406 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:49:35 dist-test-slave-rfd5 krb5kdc[16529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:49:35.293177 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35281
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:35.650179 16542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:35.650532 16542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:35.650632 16542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:35.658886 16542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:35.659044 16542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:35.659098 16542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:35.659158 16542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:35.659212 16542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:35.669790 16542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:35.671217 16542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:35.672629 16542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:35.686344 16547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:35.686316 16548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:35.686462 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:35.687067 16542 server_base.cc:1034] running on GCE node
I20260927 18:49:35.688011 16542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:35.689719 16542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:35.690971 16542 hybrid_clock.cc:648] HybridClock initialized: now 1790534975690906 us; error 66 us; skew 500 ppm
I20260927 18:49:35.691370 16542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:35 dist-test-slave-rfd5 krb5kdc[16529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534975, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:35.700682 16542 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:49:35.702647 16542 webserver.cc:460] Webserver started at http://127.31.26.254:38665/ using document root <none> and password file <none>
I20260927 18:49:35.703469 16542 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:35.703648 16542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:35.703989 16542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:35.707425 16542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "e37ca708deec4d6188df545d0bc022bb"
format_stamp: "Formatted at 2026-09-27 18:49:35 on dist-test-slave-rfd5"
server_key: "719c68c06ea8766e8e8c17889769e0d7"
server_key_iv: "160ffd0c259e122b0206857300b42c2f"
server_key_version: "encryptionkey@0"
I20260927 18:49:35.708309 16542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "e37ca708deec4d6188df545d0bc022bb"
format_stamp: "Formatted at 2026-09-27 18:49:35 on dist-test-slave-rfd5"
server_key: "719c68c06ea8766e8e8c17889769e0d7"
server_key_iv: "160ffd0c259e122b0206857300b42c2f"
server_key_version: "encryptionkey@0"
I20260927 18:49:35.714277 16542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:49:35.718421 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:35.719976 16542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:49:35.720234 16542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "e37ca708deec4d6188df545d0bc022bb"
format_stamp: "Formatted at 2026-09-27 18:49:35 on dist-test-slave-rfd5"
server_key: "719c68c06ea8766e8e8c17889769e0d7"
server_key_iv: "160ffd0c259e122b0206857300b42c2f"
server_key_version: "encryptionkey@0"
I20260927 18:49:35.720472 16542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:35.732486 16542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:35.737525 16542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:35.737861 16542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:35.758265 16542 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:35281
I20260927 18:49:35.758266 16608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:35281 every 8 connection(s)
I20260927 18:49:35.759881 16542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:35.763974 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:35.769462 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16542
I20260927 18:49:35.769717 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:35.770401 31851 external_mini_cluster.cc:1428] Setting key 5bb642ea44825c44a4a63da2bd43cafd
I20260927 18:49:35.777333 16609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb: Bootstrap starting.
I20260927 18:49:35.781669 16609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:35.782913 16609 log.cc:824] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:35.787176 16609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb: No bootstrap required, opened a new log
Sep 27 18:49:35 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:35.798393 16609 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ca708deec4d6188df545d0bc022bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35281 } }
I20260927 18:49:35.798897 16609 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:35.799033 16609 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37ca708deec4d6188df545d0bc022bb, State: Initialized, Role: FOLLOWER
I20260927 18:49:35.799898 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:35.772719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.31.26.254:35281)
0927 18:49:35.773728 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:35.773745 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:35.773804 (+ 59us) server_negotiation.cc:407] Connection header received
0927 18:49:35.776350 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:35.776386 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:35.776949 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:35.777507 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:35.779075 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:35.781248 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:35.782869 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:35.784027 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:35.784184 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:35.790134 (+ 5950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:35.790146 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:35.790169 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:35.790233 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:35.794609 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:35.795440 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:35.795455 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:35.795465 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:35.795716 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:35.796146 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:35.796151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:35.796155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:35.796998 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:35.797358 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:35.797804 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 18:49:35.798328 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":462,"threads_started":1}
I20260927 18:49:35.799793 16609 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ca708deec4d6188df545d0bc022bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35281 } }
I20260927 18:49:35.800101 16609 raft_consensus.cc:388] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:35.800256 16609 raft_consensus.cc:482] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:35.800477 16609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:35.803349 16609 raft_consensus.cc:504] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ca708deec4d6188df545d0bc022bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35281 } }
I20260927 18:49:35.803834 16609 leader_election.cc:302] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e37ca708deec4d6188df545d0bc022bb; no voters:
I20260927 18:49:35.804402 16609 leader_election.cc:288] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:35.804550 16614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:35.805403 16614 raft_consensus.cc:686] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [term 1 LEADER]: Becoming Leader. State: Replica: e37ca708deec4d6188df545d0bc022bb, State: Running, Role: LEADER
I20260927 18:49:35.806000 16614 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ca708deec4d6188df545d0bc022bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35281 } }
I20260927 18:49:35.806815 16609 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:35.809165 16615 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e37ca708deec4d6188df545d0bc022bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ca708deec4d6188df545d0bc022bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35281 } } }
I20260927 18:49:35.809267 16616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37ca708deec4d6188df545d0bc022bb. Latest consensus state: current_term: 1 leader_uuid: "e37ca708deec4d6188df545d0bc022bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37ca708deec4d6188df545d0bc022bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 35281 } } }
I20260927 18:49:35.809603 16615 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:35.809666 16616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:35.811542 16620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:35.816998 16620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:35.826436 16620 catalog_manager.cc:1347] Generated new cluster ID: 1d79baa605ca48f2b52bc2c364afd4f3
I20260927 18:49:35.826588 16620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:35.838181 16620 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:35.839670 16620 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:35.848325 16620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb: Generated new TSK 0
I20260927 18:49:35.849408 16620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:49:35.913738 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35281
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:36.278630 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:36.278971 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:36.279072 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:36.286973 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:36.287113 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:36.287231 16635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:36.297438 16635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:36.299032 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:36.300354 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:36.314929 16640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:36.314870 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:36.314968 16641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:36.315595 16635 server_base.cc:1034] running on GCE node
I20260927 18:49:36.316294 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:36.317521 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:36.318745 16635 hybrid_clock.cc:648] HybridClock initialized: now 1790534976318692 us; error 44 us; skew 500 ppm
I20260927 18:49:36.319082 16635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:36 dist-test-slave-rfd5 krb5kdc[16529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534976, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:36.322332 16635 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:49:36.323940 16635 webserver.cc:460] Webserver started at http://127.31.26.193:42365/ using document root <none> and password file <none>
I20260927 18:49:36.324705 16635 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:36.324848 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:36.325259 16635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:36.328373 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "435f31f5d19d411cb22977e1a2cfa9ec"
format_stamp: "Formatted at 2026-09-27 18:49:36 on dist-test-slave-rfd5"
server_key: "93bf7dca393671020dd18d05257f231e"
server_key_iv: "e16733cc9db12f235e75e85c600e1ee3"
server_key_version: "encryptionkey@0"
I20260927 18:49:36.329347 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "435f31f5d19d411cb22977e1a2cfa9ec"
format_stamp: "Formatted at 2026-09-27 18:49:36 on dist-test-slave-rfd5"
server_key: "93bf7dca393671020dd18d05257f231e"
server_key_iv: "e16733cc9db12f235e75e85c600e1ee3"
server_key_version: "encryptionkey@0"
I20260927 18:49:36.335294 16635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:49:36.339440 16650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:36.340909 16635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:36.341178 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "435f31f5d19d411cb22977e1a2cfa9ec"
format_stamp: "Formatted at 2026-09-27 18:49:36 on dist-test-slave-rfd5"
server_key: "93bf7dca393671020dd18d05257f231e"
server_key_iv: "e16733cc9db12f235e75e85c600e1ee3"
server_key_version: "encryptionkey@0"
I20260927 18:49:36.341415 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:36.364897 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:36.369854 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:36.370211 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:36.371393 16635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:36.372926 16635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:36.373061 16635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:36.373274 16635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:36.373350 16635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:36.409701 16635 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:35509
I20260927 18:49:36.409760 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:35509 every 8 connection(s)
I20260927 18:49:36.411136 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:36.421010 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16635
I20260927 18:49:36.421294 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:36.422004 31851 external_mini_cluster.cc:1428] Setting key b99557e0131c5b2827fba72f0f550934
Sep 27 18:49:36 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534976, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:36.435405 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.414065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:49873 (local address 127.31.26.254:35281)
0927 18:49:36.414530 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:36.414535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:36.415629 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 18:49:36.417800 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:36.417806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:36.417914 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:36.418043 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:36.419735 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.420487 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.422448 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.422919 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.423083 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:36.427485 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:36.427494 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:36.427502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:36.427551 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:36.431644 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:36.432668 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:36.432679 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:36.432685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:36.432859 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:36.433450 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:36.433457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:36.433465 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:36.433808 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:36.433927 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:36.434933 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0927 18:49:36.435132 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260927 18:49:36.437057 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.414686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35281 (local address 127.31.26.193:49873)
0927 18:49:36.415434 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:36.415483 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:36.417461 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:36.418322 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:36.418332 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:36.418913 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:36.419523 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:36.419546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.420676 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.420680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:36.422281 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:36.422296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.423249 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.423260 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:36.423615 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:36.424564 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:36.424590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:36.427191 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:36.431912 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:36.431921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:36.431946 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:36.432488 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:36.432989 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:36.432992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:36.432994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:36.433284 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:36.433943 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:36.433956 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:36.434479 (+ 523us) client_negotiation.cc:769] Sending connection context
0927 18:49:36.434869 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 18:49:36.435517 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":290,"threads_started":1}
I20260927 18:49:36.438804 16763 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35281
I20260927 18:49:36.439157 16763 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:36.439920 16763 heartbeater.cc:511] Master 127.31.26.254:35281 requested a full tablet report, sending...
I20260927 18:49:36.442256 16574 ts_manager.cc:194] Registered new tserver with Master: 435f31f5d19d411cb22977e1a2cfa9ec (127.31.26.193:35509)
I20260927 18:49:36.443758 16574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:49873
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:49:36.481144 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35281
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:36.838223 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:36.838634 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:36.838829 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:36.846880 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:36.847019 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:36.847137 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:36.858210 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:36.859601 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:36.861158 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:36.876372 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:36.876447 16774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:36.876413 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:36.876891 16769 server_base.cc:1034] running on GCE node
I20260927 18:49:36.877933 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:36.879176 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:36.880477 16769 hybrid_clock.cc:648] HybridClock initialized: now 1790534976880419 us; error 41 us; skew 500 ppm
I20260927 18:49:36.880908 16769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:36 dist-test-slave-rfd5 krb5kdc[16529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534976, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:36.884792 16769 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:49:36.886564 16769 webserver.cc:460] Webserver started at http://127.31.26.194:38849/ using document root <none> and password file <none>
I20260927 18:49:36.887332 16769 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:36.887493 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:36.887825 16769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:36.890997 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "458f8f4b7c624ade9abb87d5fe23c523"
format_stamp: "Formatted at 2026-09-27 18:49:36 on dist-test-slave-rfd5"
server_key: "a131ae7602ce2f70d48489bfe63fdd58"
server_key_iv: "5d301b31d05e309246748a9ba5620696"
server_key_version: "encryptionkey@0"
I20260927 18:49:36.891971 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "458f8f4b7c624ade9abb87d5fe23c523"
format_stamp: "Formatted at 2026-09-27 18:49:36 on dist-test-slave-rfd5"
server_key: "a131ae7602ce2f70d48489bfe63fdd58"
server_key_iv: "5d301b31d05e309246748a9ba5620696"
server_key_version: "encryptionkey@0"
I20260927 18:49:36.897846 16769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:49:36.901819 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:36.903497 16769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:36.903715 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "458f8f4b7c624ade9abb87d5fe23c523"
format_stamp: "Formatted at 2026-09-27 18:49:36 on dist-test-slave-rfd5"
server_key: "a131ae7602ce2f70d48489bfe63fdd58"
server_key_iv: "5d301b31d05e309246748a9ba5620696"
server_key_version: "encryptionkey@0"
I20260927 18:49:36.903961 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:36.932138 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:36.937052 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:36.937476 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:36.938557 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:36.940027 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:36.940140 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:36.940299 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:36.940373 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:36.976404 16769 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:34253
I20260927 18:49:36.976504 16896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:34253 every 8 connection(s)
I20260927 18:49:36.977949 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:36.987553 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16769
I20260927 18:49:36.987775 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:36.988345 31851 external_mini_cluster.cc:1428] Setting key 8b1b845c28e4055afeaea395cc15f772
Sep 27 18:49:36 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534976, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:37.002115 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.980695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:38791 (local address 127.31.26.254:35281)
0927 18:49:36.981246 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:36.981252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:36.982239 (+ 987us) server_negotiation.cc:407] Connection header received
0927 18:49:36.984326 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:36.984332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:36.984427 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:36.984539 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:36.986282 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.987037 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.989441 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.990009 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.990124 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:36.994149 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:36.994159 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:36.994167 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:36.994221 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:36.998105 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:36.999100 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:36.999110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:36.999116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:36.999318 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.000026 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.000031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.000034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.000328 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:37.000450 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.001614 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.001817 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":249,"threads_started":1}
I20260927 18:49:37.003877 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.981328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35281 (local address 127.31.26.194:38791)
0927 18:49:36.982055 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:36.982104 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:36.983936 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:36.984795 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:36.984804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:36.985458 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:36.986088 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:36.986107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.987207 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.987213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:36.989293 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:36.989314 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:36.990279 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:36.990307 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:36.990658 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:36.991419 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:36.991443 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:36.993825 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:36.998344 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:36.998354 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:36.998371 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:36.998914 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:36.999502 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:36.999510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:36.999515 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:36.999862 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:37.000482 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:37.000495 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:37.001039 (+ 544us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.001447 (+ 408us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.002132 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":273,"threads_started":1}
I20260927 18:49:37.005795 16897 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35281
I20260927 18:49:37.006170 16897 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:37.007038 16897 heartbeater.cc:511] Master 127.31.26.254:35281 requested a full tablet report, sending...
I20260927 18:49:37.008755 16574 ts_manager.cc:194] Registered new tserver with Master: 458f8f4b7c624ade9abb87d5fe23c523 (127.31.26.194:34253)
I20260927 18:49:37.009750 16574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:38791
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:49:37.049269 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35281
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:37.395421 16903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:37.395742 16903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:37.395920 16903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:37.403313 16903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:37.403448 16903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:37.403650 16903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:37.414431 16903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:37.415773 16903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:37.417209 16903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:37.431788 16908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:37.431730 16911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:37.431955 16903 server_base.cc:1034] running on GCE node
W20260927 18:49:37.431713 16909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:37.433069 16903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:37.434167 16903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:37.435421 16903 hybrid_clock.cc:648] HybridClock initialized: now 1790534977435354 us; error 70 us; skew 500 ppm
I20260927 18:49:37.435874 16903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:37 dist-test-slave-rfd5 krb5kdc[16529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534977, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:37.439260 16903 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:49:37.440855 16903 webserver.cc:460] Webserver started at http://127.31.26.195:39441/ using document root <none> and password file <none>
I20260927 18:49:37.441664 16903 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:37.441818 16903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:37.442179 16903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:37.445389 16903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "acc9ce1a7b4d4c039b1c30686504b1e6"
format_stamp: "Formatted at 2026-09-27 18:49:37 on dist-test-slave-rfd5"
server_key: "1c146e5ccda573deee5de50ad8b4b91b"
server_key_iv: "ceb918c3ceb487f4d1d5484059223900"
server_key_version: "encryptionkey@0"
I20260927 18:49:37.446266 16903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "acc9ce1a7b4d4c039b1c30686504b1e6"
format_stamp: "Formatted at 2026-09-27 18:49:37 on dist-test-slave-rfd5"
server_key: "1c146e5ccda573deee5de50ad8b4b91b"
server_key_iv: "ceb918c3ceb487f4d1d5484059223900"
server_key_version: "encryptionkey@0"
I20260927 18:49:37.447314 16763 heartbeater.cc:503] Master 127.31.26.254:35281 was elected leader, sending a full tablet report...
I20260927 18:49:37.452400 16903 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:37.456395 16918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:37.457940 16903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:37.458144 16903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "acc9ce1a7b4d4c039b1c30686504b1e6"
format_stamp: "Formatted at 2026-09-27 18:49:37 on dist-test-slave-rfd5"
server_key: "1c146e5ccda573deee5de50ad8b4b91b"
server_key_iv: "ceb918c3ceb487f4d1d5484059223900"
server_key_version: "encryptionkey@0"
I20260927 18:49:37.458384 16903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:37.476840 16903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:37.481566 16903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:37.481897 16903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:37.482977 16903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:37.484633 16903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:37.484746 16903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:37.484915 16903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:37.484990 16903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:37.520272 16903 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:34829
I20260927 18:49:37.520316 17030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:34829 every 8 connection(s)
I20260927 18:49:37.521857 16903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:37.525360 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 16903
I20260927 18:49:37.525571 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:37.526221 31851 external_mini_cluster.cc:1428] Setting key 363e4476e78f59f4c477cf20f29e9331
Sep 27 18:49:37 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534977, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:37.545197 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.524693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:38337 (local address 127.31.26.254:35281)
0927 18:49:37.525365 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.525371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.526698 (+ 1327us) server_negotiation.cc:407] Connection header received
0927 18:49:37.529200 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.529205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.529281 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.529382 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:37.531262 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.532102 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.533828 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.534248 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.534372 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.538315 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:37.538322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:37.538325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:37.538365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:37.541726 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.542813 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.542818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.542821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.542957 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.543386 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.543389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.543392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.543686 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:37.543810 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.544561 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.544828 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260927 18:49:37.546356 17034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.525463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:35281 (local address 127.31.26.195:38337)
0927 18:49:37.526442 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.526512 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:37.528747 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:37.529696 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:37.529705 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:37.530393 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:37.531062 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.531082 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.532257 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.532260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.533674 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.533689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.534464 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.534481 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.534719 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.535433 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:37.535458 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:37.538050 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:37.541955 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:37.541965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:37.541987 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:37.542639 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:37.543068 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:37.543072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:37.543074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:37.543279 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:37.543793 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:37.543799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:37.544182 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.544452 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.545017 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":343,"threads_started":1}
I20260927 18:49:37.548012 17031 heartbeater.cc:347] Connected to a master server at 127.31.26.254:35281
I20260927 18:49:37.548317 17031 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:37.549060 17031 heartbeater.cc:511] Master 127.31.26.254:35281 requested a full tablet report, sending...
I20260927 18:49:37.550686 16574 ts_manager.cc:194] Registered new tserver with Master: acc9ce1a7b4d4c039b1c30686504b1e6 (127.31.26.195:34829)
I20260927 18:49:37.551568 16574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:38337
I20260927 18:49:37.555431 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:37.574541 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.560481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.31.26.254:35281)
0927 18:49:37.560672 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.560677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.561128 (+ 451us) server_negotiation.cc:407] Connection header received
0927 18:49:37.561302 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.561306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.561375 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.561534 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:37.563218 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.564058 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.565649 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.566103 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.566422 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.569171 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:37.569187 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:37.569191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:37.569235 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:37.571835 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.572636 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.572640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.572643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.572777 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.573186 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.573189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.573191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.573530 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:37.573722 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.574113 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.574300 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:49:37.579100 16574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:37.586577 16574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:37.623095 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.600298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:34253 (local address 127.0.0.1:45172)
0927 18:49:37.607913 (+ 7615us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.607989 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:37.608273 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:37.614201 (+ 5928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:37.614210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:37.614248 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:37.614664 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.614678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.616735 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.616740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.619702 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.619715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.622058 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.622065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.622267 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.622322 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.622470 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.622690 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7216,"thread_start_us":12444,"threads_started":1}
I20260927 18:49:37.623123 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.600526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.31.26.194:34253)
0927 18:49:37.613070 (+ 12544us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.613105 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.613142 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:49:37.613270 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.613281 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.613610 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.613854 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:37.614863 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.616539 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.619904 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.621535 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.621676 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.621890 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.622562 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.622758 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12331,"thread_start_us":12776,"threads_started":1}
I20260927 18:49:37.624509 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.602202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.31.26.193:35509)
0927 18:49:37.615536 (+ 13334us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.615546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.615587 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:49:37.615691 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.615699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.615975 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.616210 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:37.616940 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.618659 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.621417 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.622931 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.623079 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.623334 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.623952 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.624139 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13150,"thread_start_us":12980,"threads_started":1}
I20260927 18:49:37.625200 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.601367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.31.26.195:34829)
0927 18:49:37.616088 (+ 14721us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.616094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.616148 (+ 54us) server_negotiation.cc:407] Connection header received
0927 18:49:37.616290 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.616301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.616571 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.616778 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:37.617918 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.619530 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.621549 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.622999 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.623164 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.623417 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.624600 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.624790 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14539,"thread_start_us":14376,"threads_started":1}
I20260927 18:49:37.625231 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.601534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:35509 (local address 127.0.0.1:32836)
0927 18:49:37.613976 (+ 12442us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.613997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:37.614113 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:37.616213 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:37.616219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:37.616253 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:37.616715 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.616732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.618812 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.618817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.621197 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.621219 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.623469 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.623474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.623730 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.623773 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.623864 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.624004 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12303,"mutex_wait_us":23,"thread_start_us":12655,"threads_started":1}
I20260927 18:49:37.625391 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.600723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:34829 (local address 127.0.0.1:46290)
0927 18:49:37.613049 (+ 12326us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.613074 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:37.613248 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:37.617054 (+ 3806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:37.617058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:37.617128 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:37.617687 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.617707 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.619703 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.619708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.621348 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.621364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.623573 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.623578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.623825 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.623855 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.623931 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.624079 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12167,"mutex_wait_us":48,"thread_start_us":315,"threads_started":1}
I20260927 18:49:37.628010 16832 tablet_service.cc:1467] Processing CreateTablet for tablet 16e9c1e67690493e91c13ab133ade974 (DEFAULT_TABLE table=test-table [id=14e5b57bc1f7483caa31790dcaa327ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:37.629379 16832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e9c1e67690493e91c13ab133ade974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:37.629297 16698 tablet_service.cc:1467] Processing CreateTablet for tablet 16e9c1e67690493e91c13ab133ade974 (DEFAULT_TABLE table=test-table [id=14e5b57bc1f7483caa31790dcaa327ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:37.629297 16966 tablet_service.cc:1467] Processing CreateTablet for tablet 16e9c1e67690493e91c13ab133ade974 (DEFAULT_TABLE table=test-table [id=14e5b57bc1f7483caa31790dcaa327ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:37.630755 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e9c1e67690493e91c13ab133ade974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:37.630750 16698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e9c1e67690493e91c13ab133ade974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:37.641317 17050 tablet_bootstrap.cc:492] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Bootstrap starting.
I20260927 18:49:37.644887 17050 tablet_bootstrap.cc:654] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:37.646520 17050 log.cc:824] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:37.646723 17051 tablet_bootstrap.cc:492] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Bootstrap starting.
I20260927 18:49:37.648958 17052 tablet_bootstrap.cc:492] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Bootstrap starting.
I20260927 18:49:37.650764 17051 tablet_bootstrap.cc:654] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:37.650949 17050 tablet_bootstrap.cc:492] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: No bootstrap required, opened a new log
I20260927 18:49:37.651240 17050 ts_tablet_manager.cc:1397] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:49:37.652070 17051 log.cc:824] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:37.652761 17052 tablet_bootstrap.cc:654] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:37.654063 17052 log.cc:824] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:37.656136 17051 tablet_bootstrap.cc:492] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: No bootstrap required, opened a new log
I20260927 18:49:37.656487 17051 ts_tablet_manager.cc:1397] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:49:37.659864 17050 raft_consensus.cc:348] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.660298 17050 raft_consensus.cc:374] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:37.660413 17050 raft_consensus.cc:729] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458f8f4b7c624ade9abb87d5fe23c523, State: Initialized, Role: FOLLOWER
I20260927 18:49:37.661197 17050 consensus_queue.cc:260] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.662499 16897 heartbeater.cc:503] Master 127.31.26.254:35281 was elected leader, sending a full tablet report...
I20260927 18:49:37.663193 17050 ts_tablet_manager.cc:1428] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 18:49:37.663964 17052 tablet_bootstrap.cc:492] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: No bootstrap required, opened a new log
I20260927 18:49:37.664330 17052 ts_tablet_manager.cc:1397] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260927 18:49:37.668906 17051 raft_consensus.cc:348] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.669428 17051 raft_consensus.cc:374] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:37.669539 17051 raft_consensus.cc:729] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc9ce1a7b4d4c039b1c30686504b1e6, State: Initialized, Role: FOLLOWER
I20260927 18:49:37.670203 17051 consensus_queue.cc:260] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.671295 17031 heartbeater.cc:503] Master 127.31.26.254:35281 was elected leader, sending a full tablet report...
I20260927 18:49:37.672040 17051 ts_tablet_manager.cc:1428] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260927 18:49:37.675688 17052 raft_consensus.cc:348] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.676101 17052 raft_consensus.cc:374] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:37.676210 17052 raft_consensus.cc:729] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435f31f5d19d411cb22977e1a2cfa9ec, State: Initialized, Role: FOLLOWER
I20260927 18:49:37.676826 17052 consensus_queue.cc:260] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.678576 17052 ts_tablet_manager.cc:1428] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260927 18:49:37.730181 16898 tablet.cc:2369] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:37.773846 17032 tablet.cc:2369] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:37.789547 17058 raft_consensus.cc:482] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:37.789893 17058 raft_consensus.cc:504] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.791723 17058 leader_election.cc:288] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 458f8f4b7c624ade9abb87d5fe23c523 (127.31.26.194:34253), acc9ce1a7b4d4c039b1c30686504b1e6 (127.31.26.195:34829)
I20260927 18:49:37.798648 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.792109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:39067 (local address 127.31.26.194:34253)
0927 18:49:37.792344 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.792350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.793265 (+ 915us) server_negotiation.cc:407] Connection header received
0927 18:49:37.793476 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.793479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.793539 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.793638 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:37.794309 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.794993 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.796828 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.797669 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.797855 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.797936 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.798229 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.798359 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:49:37.798677 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.792411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:54245 (local address 127.31.26.195:34829)
0927 18:49:37.792631 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.792637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.793179 (+ 542us) server_negotiation.cc:407] Connection header received
0927 18:49:37.793440 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.793445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.793505 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.793608 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:37.794232 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.794999 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.796891 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.797720 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.797909 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.797988 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.798278 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.798418 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:49:37.798889 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.791996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:34829 (local address 127.31.26.193:54245)
0927 18:49:37.793001 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.793026 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:37.793276 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:37.793633 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:37.793638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:37.793661 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:37.794074 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.794087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.795142 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.795149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.796724 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.796737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.797956 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.797962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.798119 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.798141 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.798240 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.798371 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":49,"thread_start_us":566,"threads_started":1}
I20260927 18:49:37.799023 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.792005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:34253 (local address 127.31.26.193:39067)
0927 18:49:37.793116 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.793158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:37.793316 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:37.793650 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:37.793654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:37.793675 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:37.794139 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.794148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.795138 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.795145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.796662 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:37.796678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.797896 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.797903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:37.798070 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.798094 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:49:37.798201 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:49:37.798345 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":1,"thread_start_us":795,"threads_started":1}
I20260927 18:49:37.800113 16852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e9c1e67690493e91c13ab133ade974" candidate_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458f8f4b7c624ade9abb87d5fe23c523" is_pre_election: true
I20260927 18:49:37.800113 16986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e9c1e67690493e91c13ab133ade974" candidate_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" is_pre_election: true
I20260927 18:49:37.800738 16986 raft_consensus.cc:2454] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435f31f5d19d411cb22977e1a2cfa9ec in term 0.
I20260927 18:49:37.800738 16852 raft_consensus.cc:2454] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435f31f5d19d411cb22977e1a2cfa9ec in term 0.
I20260927 18:49:37.801608 16652 leader_election.cc:302] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 435f31f5d19d411cb22977e1a2cfa9ec, 458f8f4b7c624ade9abb87d5fe23c523, acc9ce1a7b4d4c039b1c30686504b1e6; no voters:
I20260927 18:49:37.802098 17058 raft_consensus.cc:2781] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:37.802280 17058 raft_consensus.cc:482] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:37.802410 17058 raft_consensus.cc:3037] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:37.804679 17058 raft_consensus.cc:504] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.805399 17058 leader_election.cc:288] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [CANDIDATE]: Term 1 election: Requested vote from peers 458f8f4b7c624ade9abb87d5fe23c523 (127.31.26.194:34253), acc9ce1a7b4d4c039b1c30686504b1e6 (127.31.26.195:34829)
I20260927 18:49:37.806030 16852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e9c1e67690493e91c13ab133ade974" candidate_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458f8f4b7c624ade9abb87d5fe23c523"
I20260927 18:49:37.806094 16986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e9c1e67690493e91c13ab133ade974" candidate_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6"
I20260927 18:49:37.806311 16986 raft_consensus.cc:3037] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:37.806313 16852 raft_consensus.cc:3037] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:37.809775 16852 raft_consensus.cc:2454] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435f31f5d19d411cb22977e1a2cfa9ec in term 1.
I20260927 18:49:37.809775 16986 raft_consensus.cc:2454] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435f31f5d19d411cb22977e1a2cfa9ec in term 1.
I20260927 18:49:37.810389 16652 leader_election.cc:302] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435f31f5d19d411cb22977e1a2cfa9ec, 458f8f4b7c624ade9abb87d5fe23c523; no voters:
I20260927 18:49:37.810811 17058 raft_consensus.cc:2781] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:37.811350 17058 raft_consensus.cc:686] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 1 LEADER]: Becoming Leader. State: Replica: 435f31f5d19d411cb22977e1a2cfa9ec, State: Running, Role: LEADER
I20260927 18:49:37.811875 17058 consensus_queue.cc:237] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } }
I20260927 18:49:37.817265 16574 catalog_manager.cc:5495] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 435f31f5d19d411cb22977e1a2cfa9ec (127.31.26.193). New cstate: current_term: 1 leader_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435f31f5d19d411cb22977e1a2cfa9ec" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 35509 } health_report { overall_health: HEALTHY } } }
I20260927 18:49:37.866840 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.861168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.31.26.193:35509)
0927 18:49:37.861359 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.861364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.861382 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:49:37.861508 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.861512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.861595 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.861723 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:37.862104 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.862930 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.864554 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.864946 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.865051 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.865607 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:37.866190 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:37.866306 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.866466 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.866541 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:49:37.875617 16852 raft_consensus.cc:1261] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 1 FOLLOWER]: Refusing update from remote peer 435f31f5d19d411cb22977e1a2cfa9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:37.875622 16986 raft_consensus.cc:1261] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 1 FOLLOWER]: Refusing update from remote peer 435f31f5d19d411cb22977e1a2cfa9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:37.876626 17058 consensus_queue.cc:1035] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc9ce1a7b4d4c039b1c30686504b1e6" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 34829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:37.876950 17061 consensus_queue.cc:1035] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "458f8f4b7c624ade9abb87d5fe23c523" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:37.907328 16573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:37.907763 16573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:37.912369 16573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e37ca708deec4d6188df545d0bc022bb: Sending DeleteTablet for 3 replicas of tablet 16e9c1e67690493e91c13ab133ade974
I20260927 18:49:37.913661 16698 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e9c1e67690493e91c13ab133ade974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:37 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:32836
I20260927 18:49:37.913653 16966 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e9c1e67690493e91c13ab133ade974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:37 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:46290
I20260927 18:49:37.913730 16832 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e9c1e67690493e91c13ab133ade974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:37 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:45172
I20260927 18:49:37.918085 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16635
I20260927 18:49:37.926339 17075 tablet_replica.cc:330] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: stopping tablet replica
I20260927 18:49:37.926937 17075 raft_consensus.cc:2229] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:37.927542 17075 raft_consensus.cc:2258] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:37.928505 17074 tablet_replica.cc:330] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: stopping tablet replica
I20260927 18:49:37.928993 17074 raft_consensus.cc:2229] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:37.929780 17074 raft_consensus.cc:2258] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:37.935093 17074 ts_tablet_manager.cc:1905] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:37.939245 17075 ts_tablet_manager.cc:1905] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:37.944893 17076 tablet_replica.cc:330] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: stopping tablet replica
I20260927 18:49:37.945556 17076 raft_consensus.cc:2229] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:37.946156 17076 raft_consensus.cc:2258] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:37.950958 17075 ts_tablet_manager.cc:1918] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:37.951176 17075 log.cc:1196] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/16e9c1e67690493e91c13ab133ade974
I20260927 18:49:37.951432 17076 ts_tablet_manager.cc:1905] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:37.951809 17075 ts_tablet_manager.cc:1939] T 16e9c1e67690493e91c13ab133ade974 P acc9ce1a7b4d4c039b1c30686504b1e6: Deleting consensus metadata
Sep 27 18:49:37 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:49:37.959465 17076 ts_tablet_manager.cc:1918] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:37.959831 17076 log.cc:1196] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/16e9c1e67690493e91c13ab133ade974
I20260927 18:49:37.959810 17074 ts_tablet_manager.cc:1918] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:37.959993 17074 log.cc:1196] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/16e9c1e67690493e91c13ab133ade974
I20260927 18:49:37.960610 17074 ts_tablet_manager.cc:1939] T 16e9c1e67690493e91c13ab133ade974 P 435f31f5d19d411cb22977e1a2cfa9ec: Deleting consensus metadata
I20260927 18:49:37.960775 17076 ts_tablet_manager.cc:1939] T 16e9c1e67690493e91c13ab133ade974 P 458f8f4b7c624ade9abb87d5fe23c523: Deleting consensus metadata
I20260927 18:49:37.962921 16559 catalog_manager.cc:4841] TS 458f8f4b7c624ade9abb87d5fe23c523 (127.31.26.194:34253): tablet 16e9c1e67690493e91c13ab133ade974 (table test-table [id=14e5b57bc1f7483caa31790dcaa327ec]) successfully deleted
I20260927 18:49:37.963515 16558 catalog_manager.cc:4841] TS acc9ce1a7b4d4c039b1c30686504b1e6 (127.31.26.195:34829): tablet 16e9c1e67690493e91c13ab133ade974 (table test-table [id=14e5b57bc1f7483caa31790dcaa327ec]) successfully deleted
I20260927 18:49:37.963847 16561 catalog_manager.cc:4841] TS 435f31f5d19d411cb22977e1a2cfa9ec (127.31.26.193:35509): tablet 16e9c1e67690493e91c13ab133ade974 (table test-table [id=14e5b57bc1f7483caa31790dcaa327ec]) successfully deleted
I20260927 18:49:37.964979 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.920063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.31.26.193:35509)
0927 18:49:37.920328 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:37.920337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:37.941765 (+ 21428us) server_negotiation.cc:407] Connection header received
0927 18:49:37.942325 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:37.942331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:37.942427 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:37.942502 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:37.944339 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.945611 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.947783 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:37.948537 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:37.948690 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:37.955730 (+ 7040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:37.955752 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:37.955776 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:37.955828 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:37.960543 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.961704 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.961714 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.961718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.961925 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:37.962448 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:37.962453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:37.962456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:37.962960 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:37.963363 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:37.964330 (+ 967us) server_negotiation.cc:299] Negotiation successful
0927 18:49:37.964611 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:49:37.965821 16758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:38.080973 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16635
I20260927 18:49:38.099944 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16769
Sep 27 18:49:38 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:49:38.117285 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:38.100830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.31.26.194:34253)
0927 18:49:38.101025 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:38.101030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:38.101049 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:49:38.101175 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:38.101178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:38.101255 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:38.101340 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:38.102618 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:38.103400 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:38.104641 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:38.105157 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:38.105270 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:38.110094 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:38.110140 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:38.110164 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:38.110217 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:38.113671 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:38.114430 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:38.114438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:38.114441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:38.114584 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:38.115026 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:38.115032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:38.115035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:38.115689 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:38.116015 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:38.116429 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 18:49:38.116710 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:49:38.118240 16892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:38.212249 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16769
I20260927 18:49:38.230259 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16903
Sep 27 18:49:38 dist-test-slave-rfd5 krb5kdc[16529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:49:38.246685 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:38.231055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.31.26.195:34829)
0927 18:49:38.231245 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:38.231250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:38.231272 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:49:38.231423 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:38.231427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:38.231502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:38.231599 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:38.232959 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:38.233758 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:38.235138 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:38.235526 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:38.235624 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:38.240549 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:38.240569 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:38.240584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:38.240622 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:38.243446 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:38.244259 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:38.244268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:38.244270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:38.244503 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:38.244951 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:38.244955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:38.244957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:38.245523 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:38.245764 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:38.246124 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 18:49:38.246315 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:49:38.247360 17026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:38.339592 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16903
I20260927 18:49:38.357201 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 16542
I20260927 18:49:38.358554 16604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:38.472330 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 16542
2026-09-27T18:49:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4564 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T18:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:38Z Disabled control of system clock
I20260927 18:49:38.555572 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42067
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:49:38.905306 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:38.905615 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:38.905706 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:38.913066 17085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:38.913255 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:38.913352 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:38.913460 17085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:38.913552 17085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:38.923074 17085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:42067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:38.924467 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:38.926419 17085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:38.938535 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:38.938540 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:38.938581 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:38.939620 17085 server_base.cc:1034] running on GCE node
I20260927 18:49:38.940538 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:38.942147 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:38.943372 17085 hybrid_clock.cc:648] HybridClock initialized: now 1790534978943328 us; error 28 us; skew 500 ppm
I20260927 18:49:38.943797 17085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:38.946496 17085 webserver.cc:460] Webserver started at http://127.31.26.254:39795/ using document root <none> and password file <none>
I20260927 18:49:38.947269 17085 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:38.947368 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:38.947729 17085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:38.951014 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "25e5a44e6caa428a8b7785c5aec6aa05"
format_stamp: "Formatted at 2026-09-27 18:49:38 on dist-test-slave-rfd5"
server_key: "eb13e1e5c7cdadbb8f7c0462ad00376e"
server_key_iv: "04ac1575d26e40373da11c268cd2421f"
server_key_version: "encryptionkey@0"
I20260927 18:49:38.951948 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "25e5a44e6caa428a8b7785c5aec6aa05"
format_stamp: "Formatted at 2026-09-27 18:49:38 on dist-test-slave-rfd5"
server_key: "eb13e1e5c7cdadbb8f7c0462ad00376e"
server_key_iv: "04ac1575d26e40373da11c268cd2421f"
server_key_version: "encryptionkey@0"
I20260927 18:49:38.957816 17085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 18:49:38.961990 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:38.963593 17085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:38.963829 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "25e5a44e6caa428a8b7785c5aec6aa05"
format_stamp: "Formatted at 2026-09-27 18:49:38 on dist-test-slave-rfd5"
server_key: "eb13e1e5c7cdadbb8f7c0462ad00376e"
server_key_iv: "04ac1575d26e40373da11c268cd2421f"
server_key_version: "encryptionkey@0"
I20260927 18:49:38.964114 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:38.983208 17085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:38.984161 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:38.984478 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:39.004942 17085 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:42067
I20260927 18:49:39.004962 17150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:42067 every 8 connection(s)
I20260927 18:49:39.006522 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:39.010706 17151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:39.011703 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17085
I20260927 18:49:39.011901 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:39.012364 31851 external_mini_cluster.cc:1428] Setting key c139cbcfede78791a5562e48872a1d44
I20260927 18:49:39.026958 17151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05: Bootstrap starting.
I20260927 18:49:39.029407 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:39.014515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.31.26.254:42067)
0927 18:49:39.015832 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:39.015847 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:39.015908 (+ 61us) server_negotiation.cc:407] Connection header received
0927 18:49:39.018026 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:39.018063 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:39.018826 (+ 763us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:39.019377 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:39.019933 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:39.021899 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:39.023439 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:39.024685 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:39.024895 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:39.026194 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:39.026206 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:39.026231 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:39.027041 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:39.027163 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:39.027191 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:39.027319 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:39.027548 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:49:39.027928 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":436,"threads_started":1}
I20260927 18:49:39.030999 17151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:39.032119 17151 log.cc:824] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:39.035274 17151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05: No bootstrap required, opened a new log
I20260927 18:49:39.043573 17151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42067 } }
I20260927 18:49:39.044026 17151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:39.044148 17151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e5a44e6caa428a8b7785c5aec6aa05, State: Initialized, Role: FOLLOWER
I20260927 18:49:39.044781 17151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42067 } }
I20260927 18:49:39.045075 17151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:39.045295 17151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:39.045506 17151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:39.048328 17151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42067 } }
I20260927 18:49:39.048909 17151 leader_election.cc:302] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e5a44e6caa428a8b7785c5aec6aa05; no voters:
I20260927 18:49:39.049563 17151 leader_election.cc:288] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:39.049748 17156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:39.050609 17156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [term 1 LEADER]: Becoming Leader. State: Replica: 25e5a44e6caa428a8b7785c5aec6aa05, State: Running, Role: LEADER
I20260927 18:49:39.051178 17156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42067 } }
I20260927 18:49:39.052052 17151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:39.054497 17157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42067 } } }
I20260927 18:49:39.055064 17157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:39.054658 17158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e5a44e6caa428a8b7785c5aec6aa05. Latest consensus state: current_term: 1 leader_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5a44e6caa428a8b7785c5aec6aa05" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42067 } } }
I20260927 18:49:39.055298 17158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:39.057343 17165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:39.062281 17165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:39.072045 17165 catalog_manager.cc:1347] Generated new cluster ID: c830dbf113594b6fa76c6feca7d78427
I20260927 18:49:39.072341 17165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:39.082566 17165 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:39.084597 17165 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:39.098119 17165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05: Generated new TSK 0
I20260927 18:49:39.099136 17165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:49:39.106767 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42067
--builtin_ntp_servers=127.31.26.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:49:39.460309 17175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:39.460620 17175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:39.460794 17175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:39.468525 17175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:39.468676 17175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:39.468854 17175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:39.478593 17175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:39.479974 17175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:39.481897 17175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:39.496268 17181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:39.496215 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:39.496316 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:39.498847 17175 server_base.cc:1034] running on GCE node
I20260927 18:49:39.499600 17175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:39.500644 17175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:39.501852 17175 hybrid_clock.cc:648] HybridClock initialized: now 1790534979501794 us; error 49 us; skew 500 ppm
I20260927 18:49:39.502200 17175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:39.504659 17175 webserver.cc:460] Webserver started at http://127.31.26.193:33925/ using document root <none> and password file <none>
I20260927 18:49:39.505492 17175 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:39.505625 17175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:39.505959 17175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:39.509027 17175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "53592cdd7bd746bd8b41acb2bbae1ffb"
format_stamp: "Formatted at 2026-09-27 18:49:39 on dist-test-slave-rfd5"
server_key: "0fadcfb691fb9f2a39bec3a019ff2991"
server_key_iv: "f3664e01504236e329e6cbd9a7d81cb4"
server_key_version: "encryptionkey@0"
I20260927 18:49:39.509936 17175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "53592cdd7bd746bd8b41acb2bbae1ffb"
format_stamp: "Formatted at 2026-09-27 18:49:39 on dist-test-slave-rfd5"
server_key: "0fadcfb691fb9f2a39bec3a019ff2991"
server_key_iv: "f3664e01504236e329e6cbd9a7d81cb4"
server_key_version: "encryptionkey@0"
I20260927 18:49:39.515287 17175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:49:39.519069 17189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:39.520432 17175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:39.520637 17175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "53592cdd7bd746bd8b41acb2bbae1ffb"
format_stamp: "Formatted at 2026-09-27 18:49:39 on dist-test-slave-rfd5"
server_key: "0fadcfb691fb9f2a39bec3a019ff2991"
server_key_iv: "f3664e01504236e329e6cbd9a7d81cb4"
server_key_version: "encryptionkey@0"
I20260927 18:49:39.520874 17175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:39.535714 17175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:39.536679 17175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:39.537002 17175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:39.538064 17175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:39.539587 17175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:39.539722 17175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:39.539856 17175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:39.539943 17175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:39.583438 17175 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:36027
I20260927 18:49:39.583506 17301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:36027 every 8 connection(s)
I20260927 18:49:39.585430 17175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:39.593170 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17175
I20260927 18:49:39.593389 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:39.593995 31851 external_mini_cluster.cc:1428] Setting key 2587e59cbbd1b5001394e98a33d503bb
I20260927 18:49:39.597317 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42067
--builtin_ntp_servers=127.31.26.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:49:39.603008 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:39.588640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:33363 (local address 127.31.26.254:42067)
0927 18:49:39.589171 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:39.589178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:39.591185 (+ 2007us) server_negotiation.cc:407] Connection header received
0927 18:49:39.593748 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:39.593755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:39.593853 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:39.593994 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:39.595567 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:39.596860 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:39.599336 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:39.600123 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:39.600269 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:39.602013 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:39.602022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:39.602029 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:39.602058 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:39.602122 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:39.602143 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:39.602239 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:39.602520 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 18:49:39.602644 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260927 18:49:39.604379 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:39.589607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42067 (local address 127.31.26.193:33363)
0927 18:49:39.590955 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:39.591030 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:39.593348 (+ 2318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:39.594293 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:39.594311 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:39.594480 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:39.595370 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:39.595397 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:39.597113 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:39.597122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:39.599173 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:39.599188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:39.600396 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:39.600405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:39.600709 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:39.601545 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:39.601551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:39.601616 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:39.601736 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:39.602252 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:39.602267 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:39.602276 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:49:39.602492 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 18:49:39.603032 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":411,"threads_started":1}
I20260927 18:49:39.609242 17302 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42067
I20260927 18:49:39.609671 17302 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:39.610834 17302 heartbeater.cc:511] Master 127.31.26.254:42067 requested a full tablet report, sending...
I20260927 18:49:39.613853 17116 ts_manager.cc:194] Registered new tserver with Master: 53592cdd7bd746bd8b41acb2bbae1ffb (127.31.26.193:36027)
I20260927 18:49:39.616145 17116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.193:33363
W20260927 18:49:39.945494 17306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:39.945887 17306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:39.946070 17306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:39.957053 17306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:39.957228 17306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:39.957419 17306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:39.967345 17306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:39.968812 17306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:39.970711 17306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:39.984277 17311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:39.984212 17312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:39.984301 17314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:39.985555 17306 server_base.cc:1034] running on GCE node
I20260927 18:49:39.986158 17306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:39.987135 17306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:39.988353 17306 hybrid_clock.cc:648] HybridClock initialized: now 1790534979988286 us; error 65 us; skew 500 ppm
I20260927 18:49:39.988680 17306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:39.991196 17306 webserver.cc:460] Webserver started at http://127.31.26.194:41107/ using document root <none> and password file <none>
I20260927 18:49:39.991887 17306 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:39.991977 17306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:39.992236 17306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:39.995216 17306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "9d0a11abed56413996b291431e9d03cf"
format_stamp: "Formatted at 2026-09-27 18:49:39 on dist-test-slave-rfd5"
server_key: "c1c9b491140113e45be9196e4f85bd2f"
server_key_iv: "3cedffea4cbcffeb3ca13fa7b76e16d3"
server_key_version: "encryptionkey@0"
I20260927 18:49:39.995994 17306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "9d0a11abed56413996b291431e9d03cf"
format_stamp: "Formatted at 2026-09-27 18:49:39 on dist-test-slave-rfd5"
server_key: "c1c9b491140113e45be9196e4f85bd2f"
server_key_iv: "3cedffea4cbcffeb3ca13fa7b76e16d3"
server_key_version: "encryptionkey@0"
I20260927 18:49:40.001502 17306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:49:40.005443 17320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:40.007150 17306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:49:40.007325 17306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "9d0a11abed56413996b291431e9d03cf"
format_stamp: "Formatted at 2026-09-27 18:49:39 on dist-test-slave-rfd5"
server_key: "c1c9b491140113e45be9196e4f85bd2f"
server_key_iv: "3cedffea4cbcffeb3ca13fa7b76e16d3"
server_key_version: "encryptionkey@0"
I20260927 18:49:40.007570 17306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:40.028534 17306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:40.029434 17306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:40.029696 17306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:40.030674 17306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:40.032159 17306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:40.032249 17306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:40.032326 17306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:40.032383 17306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:40.066586 17306 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:41197
I20260927 18:49:40.066704 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:41197 every 8 connection(s)
I20260927 18:49:40.067976 17306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:40.069200 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17306
I20260927 18:49:40.069393 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:40.069936 31851 external_mini_cluster.cc:1428] Setting key ebe39ebb3e2b39ce71c3334465af9705
I20260927 18:49:40.073315 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42067
--builtin_ntp_servers=127.31.26.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:49:40.081344 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.070893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:35235 (local address 127.31.26.254:42067)
0927 18:49:40.071063 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.071067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.072792 (+ 1725us) server_negotiation.cc:407] Connection header received
0927 18:49:40.074831 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.074837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.074938 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.075053 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:40.076213 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.076930 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.078547 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.078954 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.079096 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.080519 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:40.080524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:40.080527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:40.080546 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:40.080577 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:40.080589 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:40.080686 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.080903 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.081005 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:49:40.082446 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.071562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42067 (local address 127.31.26.194:35235)
0927 18:49:40.072640 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.072678 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.074532 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.075311 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.075319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.075425 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:40.076042 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.076060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.077127 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.077134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.078398 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.078413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.079183 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.079189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.079435 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.080194 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:40.080196 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:40.080236 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:40.080317 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:40.080683 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:40.080695 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:40.080704 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.080902 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.081347 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":390,"threads_started":1}
I20260927 18:49:40.083959 17433 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42067
I20260927 18:49:40.084288 17433 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:40.085291 17433 heartbeater.cc:511] Master 127.31.26.254:42067 requested a full tablet report, sending...
I20260927 18:49:40.087049 17116 ts_manager.cc:194] Registered new tserver with Master: 9d0a11abed56413996b291431e9d03cf (127.31.26.194:41197)
I20260927 18:49:40.088352 17116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.194:35235
W20260927 18:49:40.421532 17436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:40.421832 17436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:40.421924 17436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:40.429557 17436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:40.429673 17436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:40.429780 17436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:40.439481 17436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:40.440768 17436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:40.442550 17436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:40.456247 17442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:40.456369 17444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:40.456835 17441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:40.457391 17436 server_base.cc:1034] running on GCE node
I20260927 18:49:40.458072 17436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:40.459143 17436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:40.460376 17436 hybrid_clock.cc:648] HybridClock initialized: now 1790534980460305 us; error 69 us; skew 500 ppm
I20260927 18:49:40.460721 17436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:40.463251 17436 webserver.cc:460] Webserver started at http://127.31.26.195:39887/ using document root <none> and password file <none>
I20260927 18:49:40.464002 17436 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:40.464133 17436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:40.464473 17436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:40.467579 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "cf3f13f897d744cebd83a338582da88d"
format_stamp: "Formatted at 2026-09-27 18:49:40 on dist-test-slave-rfd5"
server_key: "90b86ecb74edcfb2885e4d1a508215a4"
server_key_iv: "7bcf8370ae65aacc2f1afca0dfe044ed"
server_key_version: "encryptionkey@0"
I20260927 18:49:40.468538 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "cf3f13f897d744cebd83a338582da88d"
format_stamp: "Formatted at 2026-09-27 18:49:40 on dist-test-slave-rfd5"
server_key: "90b86ecb74edcfb2885e4d1a508215a4"
server_key_iv: "7bcf8370ae65aacc2f1afca0dfe044ed"
server_key_version: "encryptionkey@0"
I20260927 18:49:40.474176 17436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 18:49:40.477983 17450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:40.479384 17436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:40.479588 17436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "cf3f13f897d744cebd83a338582da88d"
format_stamp: "Formatted at 2026-09-27 18:49:40 on dist-test-slave-rfd5"
server_key: "90b86ecb74edcfb2885e4d1a508215a4"
server_key_iv: "7bcf8370ae65aacc2f1afca0dfe044ed"
server_key_version: "encryptionkey@0"
I20260927 18:49:40.479835 17436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:40.504688 17436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:40.505661 17436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:40.505983 17436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:40.506989 17436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:40.508476 17436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:40.508591 17436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:40.508723 17436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:40.508816 17436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:40.544386 17436 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:46003
I20260927 18:49:40.544449 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:46003 every 8 connection(s)
I20260927 18:49:40.545843 17436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:40.551244 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17436
I20260927 18:49:40.551451 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:40.551981 31851 external_mini_cluster.cc:1428] Setting key ba9244e15ec7e598a27467307aa83f8e
I20260927 18:49:40.559427 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.548507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:59139 (local address 127.31.26.254:42067)
0927 18:49:40.548737 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.548745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.550248 (+ 1503us) server_negotiation.cc:407] Connection header received
0927 18:49:40.552368 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.552375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.552473 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.552606 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:40.553964 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.554749 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.556501 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.556904 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.557056 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.558564 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:40.558569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:40.558572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:40.558593 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:40.558626 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:40.558638 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:40.558740 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.558982 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.559098 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 18:49:40.560529 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.549040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42067 (local address 127.31.26.195:59139)
0927 18:49:40.550050 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.550103 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.551989 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.552912 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.552931 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.553043 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:40.553778 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.553797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.554911 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.554917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.556300 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.556312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.557187 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.557195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.557446 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.558208 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:40.558211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:40.558253 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:40.558342 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:40.558737 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:40.558749 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:40.558759 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.558951 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.559418 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":270,"threads_started":1}
I20260927 18:49:40.562173 17563 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42067
I20260927 18:49:40.562522 17563 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:40.563313 17563 heartbeater.cc:511] Master 127.31.26.254:42067 requested a full tablet report, sending...
I20260927 18:49:40.564952 17116 ts_manager.cc:194] Registered new tserver with Master: cf3f13f897d744cebd83a338582da88d (127.31.26.195:46003)
I20260927 18:49:40.565994 17116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.195:59139
I20260927 18:49:40.567732 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:40.578685 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.572245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.31.26.254:42067)
0927 18:49:40.572494 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.572499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.572954 (+ 455us) server_negotiation.cc:407] Connection header received
0927 18:49:40.573177 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.573181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.573272 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.573373 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:40.573933 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.574697 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.576191 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.576630 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.576729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.577990 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:40.577995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:40.577999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:40.578020 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:40.578057 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:40.578070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:40.578146 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.578324 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.578419 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:49:40.582870 17116 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:40.590126 17116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:40.615751 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.603614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.31.26.195:46003)
0927 18:49:40.604891 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.604901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.606855 (+ 1954us) server_negotiation.cc:407] Connection header received
0927 18:49:40.607262 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.607271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.607579 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.607786 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:40.608847 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.610542 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.613268 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.614417 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.614591 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.614921 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.615189 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.615360 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":943,"threads_started":1}
I20260927 18:49:40.615782 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.603458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:46003 (local address 127.0.0.1:48932)
0927 18:49:40.606714 (+ 3256us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.606736 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.606989 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.608192 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.608197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.608223 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:40.608648 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.608658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.610691 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.610697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.612965 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.612979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.614653 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.614660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.614899 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.614967 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.615132 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.615345 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3046,"mutex_wait_us":66,"thread_start_us":1669,"threads_started":1}
I20260927 18:49:40.616428 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.602908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:36027 (local address 127.0.0.1:32848)
0927 18:49:40.605642 (+ 2734us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.605697 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.605952 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.608152 (+ 2200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.608164 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.608209 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:40.608904 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.608920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.610414 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.610425 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.613253 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.613267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.615618 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.615626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.615799 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.615833 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.615919 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.616056 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2381,"mutex_wait_us":152,"thread_start_us":3609,"threads_started":1}
I20260927 18:49:40.617341 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.603233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.31.26.193:36027)
0927 18:49:40.604507 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.604532 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.607372 (+ 2840us) server_negotiation.cc:407] Connection header received
0927 18:49:40.607481 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.607489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.607696 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.608102 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:40.609148 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.610260 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.613494 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.615001 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.615176 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.615462 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.616751 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.616942 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":949,"threads_started":1}
I20260927 18:49:40.619077 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.604310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:41197 (local address 127.0.0.1:58520)
0927 18:49:40.606815 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.606835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.606955 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.610175 (+ 3220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.610181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.610208 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:40.610614 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.610901 (+ 287us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.612992 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.612998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.616616 (+ 3618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.616633 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.618330 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.618337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.618506 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.618538 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.618636 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.618770 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2383,"thread_start_us":1998,"threads_started":1}
I20260927 18:49:40.619221 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.604951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.31.26.194:41197)
0927 18:49:40.609267 (+ 4316us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.609276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.609309 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:49:40.609419 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.609428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.609729 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.609926 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:40.611130 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.612817 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.616836 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.618125 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.618298 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.618553 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.618686 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.618845 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4145,"thread_start_us":4009,"threads_started":1}
I20260927 18:49:40.620743 17302 heartbeater.cc:503] Master 127.31.26.254:42067 was elected leader, sending a full tablet report...
I20260927 18:49:40.621155 17498 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 (DEFAULT_TABLE table=test-table [id=75131bec8b7440e39cef1096ee872a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:40.621078 17237 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 (DEFAULT_TABLE table=test-table [id=75131bec8b7440e39cef1096ee872a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:40.622740 17498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:40.622735 17237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:40.622889 17368 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 (DEFAULT_TABLE table=test-table [id=75131bec8b7440e39cef1096ee872a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:40.624094 17368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:40.636322 17581 tablet_bootstrap.cc:492] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Bootstrap starting.
I20260927 18:49:40.639670 17581 tablet_bootstrap.cc:654] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:40.640641 17583 tablet_bootstrap.cc:492] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Bootstrap starting.
I20260927 18:49:40.641079 17581 log.cc:824] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:40.643968 17582 tablet_bootstrap.cc:492] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Bootstrap starting.
I20260927 18:49:40.645727 17583 tablet_bootstrap.cc:654] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:40.647396 17583 log.cc:824] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:40.647979 17582 tablet_bootstrap.cc:654] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:40.648974 17582 log.cc:824] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:40.652073 17583 tablet_bootstrap.cc:492] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: No bootstrap required, opened a new log
I20260927 18:49:40.652388 17583 ts_tablet_manager.cc:1397] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 18:49:40.652542 17581 tablet_bootstrap.cc:492] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: No bootstrap required, opened a new log
I20260927 18:49:40.652843 17581 ts_tablet_manager.cc:1397] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20260927 18:49:40.655117 17582 tablet_bootstrap.cc:492] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: No bootstrap required, opened a new log
I20260927 18:49:40.655464 17582 ts_tablet_manager.cc:1397] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 18:49:40.663362 17582 raft_consensus.cc:348] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.663914 17582 raft_consensus.cc:374] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:40.664028 17582 raft_consensus.cc:729] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3f13f897d744cebd83a338582da88d, State: Initialized, Role: FOLLOWER
I20260927 18:49:40.664040 17581 raft_consensus.cc:348] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.664040 17583 raft_consensus.cc:348] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.664495 17583 raft_consensus.cc:374] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:40.664494 17581 raft_consensus.cc:374] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:40.664599 17581 raft_consensus.cc:729] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a11abed56413996b291431e9d03cf, State: Initialized, Role: FOLLOWER
I20260927 18:49:40.664599 17583 raft_consensus.cc:729] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53592cdd7bd746bd8b41acb2bbae1ffb, State: Initialized, Role: FOLLOWER
I20260927 18:49:40.664815 17582 consensus_queue.cc:260] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.665243 17583 consensus_queue.cc:260] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.665243 17581 consensus_queue.cc:260] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.666354 17563 heartbeater.cc:503] Master 127.31.26.254:42067 was elected leader, sending a full tablet report...
I20260927 18:49:40.667187 17582 ts_tablet_manager.cc:1428] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 18:49:40.674067 17583 ts_tablet_manager.cc:1428] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s
I20260927 18:49:40.675282 17433 heartbeater.cc:503] Master 127.31.26.254:42067 was elected leader, sending a full tablet report...
I20260927 18:49:40.675935 17581 ts_tablet_manager.cc:1428] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Time spent starting tablet: real 0.023s user 0.016s sys 0.008s
I20260927 18:49:40.754778 17587 raft_consensus.cc:482] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:40.755088 17587 raft_consensus.cc:504] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.757140 17587 leader_election.cc:288] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53592cdd7bd746bd8b41acb2bbae1ffb (127.31.26.193:36027), 9d0a11abed56413996b291431e9d03cf (127.31.26.194:41197)
I20260927 18:49:40.765194 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.757508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:41197 (local address 127.31.26.195:34731)
0927 18:49:40.757867 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.757895 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.758095 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.758550 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.758554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.758620 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:40.758981 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.758999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.760447 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.760453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.762810 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.762828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.764412 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.764417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.764546 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.764584 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.764662 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.764774 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"spinlock_wait_cycles":6016}
I20260927 18:49:40.765252 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.757651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:34731 (local address 127.31.26.194:41197)
0927 18:49:40.757871 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.757880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.758040 (+ 160us) server_negotiation.cc:407] Connection header received
0927 18:49:40.758277 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.758282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.758387 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.758514 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:40.759196 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.760295 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.763034 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.764236 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.764432 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.764517 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.764738 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.764898 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:49:40.766057 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.757751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:47751 (local address 127.31.26.193:36027)
0927 18:49:40.757971 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.757980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.758646 (+ 666us) server_negotiation.cc:407] Connection header received
0927 18:49:40.758836 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.758843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.758948 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.759068 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:40.759785 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.760840 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.763428 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.764740 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.764914 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.764990 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.765597 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.765747 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:49:40.766076 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.757597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:36027 (local address 127.31.26.195:47751)
0927 18:49:40.758514 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.758534 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:40.758671 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:40.759097 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:40.759101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:40.759130 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:40.759564 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.759578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.761006 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.761012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.763205 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:40.763216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.764936 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.764943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:40.765155 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.765201 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:49:40.765543 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 18:49:40.765691 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":1,"thread_start_us":465,"threads_started":1}
I20260927 18:49:40.766574 17388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbfd28dca7a4a01b25cbc8fc947a2f3" candidate_uuid: "cf3f13f897d744cebd83a338582da88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a11abed56413996b291431e9d03cf" is_pre_election: true
I20260927 18:49:40.767215 17388 raft_consensus.cc:2454] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf3f13f897d744cebd83a338582da88d in term 0.
I20260927 18:49:40.767329 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbfd28dca7a4a01b25cbc8fc947a2f3" candidate_uuid: "cf3f13f897d744cebd83a338582da88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" is_pre_election: true
I20260927 18:49:40.767968 17257 raft_consensus.cc:2454] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf3f13f897d744cebd83a338582da88d in term 0.
I20260927 18:49:40.768044 17454 leader_election.cc:302] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a11abed56413996b291431e9d03cf, cf3f13f897d744cebd83a338582da88d; no voters:
I20260927 18:49:40.768543 17587 raft_consensus.cc:2781] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:40.768738 17587 raft_consensus.cc:482] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:40.768898 17587 raft_consensus.cc:3037] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:40.771504 17587 raft_consensus.cc:504] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.772156 17587 leader_election.cc:288] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [CANDIDATE]: Term 1 election: Requested vote from peers 53592cdd7bd746bd8b41acb2bbae1ffb (127.31.26.193:36027), 9d0a11abed56413996b291431e9d03cf (127.31.26.194:41197)
I20260927 18:49:40.772691 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbfd28dca7a4a01b25cbc8fc947a2f3" candidate_uuid: "cf3f13f897d744cebd83a338582da88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb"
I20260927 18:49:40.772706 17388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbfd28dca7a4a01b25cbc8fc947a2f3" candidate_uuid: "cf3f13f897d744cebd83a338582da88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a11abed56413996b291431e9d03cf"
I20260927 18:49:40.772897 17257 raft_consensus.cc:3037] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:40.772944 17388 raft_consensus.cc:3037] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:40.775231 17257 raft_consensus.cc:2454] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf3f13f897d744cebd83a338582da88d in term 1.
I20260927 18:49:40.775612 17388 raft_consensus.cc:2454] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf3f13f897d744cebd83a338582da88d in term 1.
I20260927 18:49:40.775727 17453 leader_election.cc:302] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53592cdd7bd746bd8b41acb2bbae1ffb, cf3f13f897d744cebd83a338582da88d; no voters:
I20260927 18:49:40.776142 17587 raft_consensus.cc:2781] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:40.776707 17587 raft_consensus.cc:686] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 1 LEADER]: Becoming Leader. State: Replica: cf3f13f897d744cebd83a338582da88d, State: Running, Role: LEADER
I20260927 18:49:40.777370 17587 consensus_queue.cc:237] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } }
I20260927 18:49:40.782464 17115 catalog_manager.cc:5495] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d reported cstate change: term changed from 0 to 1, leader changed from <none> to cf3f13f897d744cebd83a338582da88d (127.31.26.195). New cstate: current_term: 1 leader_uuid: "cf3f13f897d744cebd83a338582da88d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf3f13f897d744cebd83a338582da88d" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 46003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 } health_report { overall_health: UNKNOWN } } }
W20260927 18:49:40.797621 17564 tablet.cc:2369] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:40.810472 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.804639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.31.26.195:46003)
0927 18:49:40.804873 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.804879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.804923 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:49:40.805158 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.805164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.805261 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.805403 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:40.805754 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.806677 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.808083 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.808536 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.808641 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.809259 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:40.809800 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:40.809923 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.810066 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.810139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:49:40.819118 17257 raft_consensus.cc:1261] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 1 FOLLOWER]: Refusing update from remote peer cf3f13f897d744cebd83a338582da88d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:40.819106 17388 raft_consensus.cc:1261] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 1 FOLLOWER]: Refusing update from remote peer cf3f13f897d744cebd83a338582da88d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:40.820144 17587 consensus_queue.cc:1035] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53592cdd7bd746bd8b41acb2bbae1ffb" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:49:40.820341 17434 tablet.cc:2369] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:40.820809 17587 consensus_queue.cc:1035] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a11abed56413996b291431e9d03cf" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:40.831133 17594 mvcc.cc:205] Tried to move back new op lower bound from 7334031281424543744 to 7334031281268781056. Current Snapshot: MvccSnapshot[applied={T|T < 7334031281424543744}]
I20260927 18:49:40.846634 17115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:40.846971 17115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:40.851049 17115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25e5a44e6caa428a8b7785c5aec6aa05: Sending DeleteTablet for 3 replicas of tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3
I20260927 18:49:40.852411 17237 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:40 UTC) from {username='slave'} at 127.0.0.1:32848
I20260927 18:49:40.852423 17498 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:40 UTC) from {username='slave'} at 127.0.0.1:48932
I20260927 18:49:40.852481 17368 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:40 UTC) from {username='slave'} at 127.0.0.1:58520
I20260927 18:49:40.860566 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17175
I20260927 18:49:40.864084 17605 tablet_replica.cc:330] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: stopping tablet replica
I20260927 18:49:40.864636 17605 raft_consensus.cc:2229] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:40.865535 17605 raft_consensus.cc:2258] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:40.866410 17604 tablet_replica.cc:330] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: stopping tablet replica
I20260927 18:49:40.867131 17604 raft_consensus.cc:2229] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:40.867703 17604 raft_consensus.cc:2258] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:40.869103 17603 tablet_replica.cc:330] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: stopping tablet replica
I20260927 18:49:40.869602 17603 raft_consensus.cc:2229] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:40.870188 17603 raft_consensus.cc:2258] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:40.870393 17604 ts_tablet_manager.cc:1905] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:40.872581 17603 ts_tablet_manager.cc:1905] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:40.875321 17605 ts_tablet_manager.cc:1905] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:40.878008 17604 ts_tablet_manager.cc:1918] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:40.878211 17604 log.cc:1196] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/5dbfd28dca7a4a01b25cbc8fc947a2f3
I20260927 18:49:40.878813 17604 ts_tablet_manager.cc:1939] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 53592cdd7bd746bd8b41acb2bbae1ffb: Deleting consensus metadata
I20260927 18:49:40.880302 17603 ts_tablet_manager.cc:1918] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:40.880569 17603 log.cc:1196] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/5dbfd28dca7a4a01b25cbc8fc947a2f3
I20260927 18:49:40.880968 17102 catalog_manager.cc:4841] TS 53592cdd7bd746bd8b41acb2bbae1ffb (127.31.26.193:36027): tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 (table test-table [id=75131bec8b7440e39cef1096ee872a83]) successfully deleted
I20260927 18:49:40.881336 17603 ts_tablet_manager.cc:1939] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P 9d0a11abed56413996b291431e9d03cf: Deleting consensus metadata
I20260927 18:49:40.883496 17605 ts_tablet_manager.cc:1918] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:40.883481 17103 catalog_manager.cc:4841] TS 9d0a11abed56413996b291431e9d03cf (127.31.26.194:41197): tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 (table test-table [id=75131bec8b7440e39cef1096ee872a83]) successfully deleted
I20260927 18:49:40.883724 17605 log.cc:1196] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/5dbfd28dca7a4a01b25cbc8fc947a2f3
I20260927 18:49:40.884286 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.862832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.31.26.193:36027)
0927 18:49:40.863073 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:40.863083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:40.866767 (+ 3684us) server_negotiation.cc:407] Connection header received
0927 18:49:40.867147 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:40.867157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:40.867304 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:40.867421 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:40.868320 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.869545 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.871625 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:40.872313 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:40.872422 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:40.882082 (+ 9660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:40.882095 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:40.882113 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:40.882918 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:40.882988 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:40.883018 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:40.883143 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:40.883769 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 18:49:40.883905 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:49:40.884367 17605 ts_tablet_manager.cc:1939] T 5dbfd28dca7a4a01b25cbc8fc947a2f3 P cf3f13f897d744cebd83a338582da88d: Deleting consensus metadata
I20260927 18:49:40.885051 17297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:40.886246 17100 catalog_manager.cc:4841] TS cf3f13f897d744cebd83a338582da88d (127.31.26.195:46003): tablet 5dbfd28dca7a4a01b25cbc8fc947a2f3 (table test-table [id=75131bec8b7440e39cef1096ee872a83]) successfully deleted
I20260927 18:49:40.979009 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17175
I20260927 18:49:41.003798 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17306
I20260927 18:49:41.011277 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:41.004741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.31.26.194:41197)
0927 18:49:41.004935 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:41.004942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:41.005053 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:49:41.005306 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:41.005310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:41.005399 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:41.005517 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:41.005946 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:41.006712 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:41.008045 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:41.008462 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:41.008605 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:41.009770 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:41.009779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:41.009794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:41.010413 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:41.010464 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:41.010484 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:41.010613 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:41.010747 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:49:41.010874 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:49:41.012063 17428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:41.110605 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17306
I20260927 18:49:41.128281 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17436
I20260927 18:49:41.136016 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:41.129268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.31.26.195:46003)
0927 18:49:41.129480 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:41.129486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:41.129507 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:49:41.129644 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:41.129652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:41.129752 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:41.129933 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:41.130462 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:41.131449 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:41.132709 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:41.133232 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:41.133391 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:41.134536 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:41.134546 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:41.134560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:41.135189 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:41.135247 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:41.135270 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:41.135412 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:41.135543 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 18:49:41.135643 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:49:41.136827 17558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:41.260450 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17436
I20260927 18:49:41.279884 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17085
I20260927 18:49:41.281175 17146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:41.392060 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17085
2026-09-27T18:49:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2913 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:41 dist-test-slave-rfd5 krb5kdc[17613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:41 dist-test-slave-rfd5 krb5kdc[17613](info): set up 2 sockets
Sep 27 18:49:41 dist-test-slave-rfd5 krb5kdc[17613](info): commencing operation
krb5kdc: starting...
W20260927 18:49:42.589723 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:49:42 dist-test-slave-rfd5 krb5kdc[17613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:49:42.852877 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42457
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:42457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:43.195524 17626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:43.195931 17626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:43.196105 17626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:43.204020 17626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:43.204181 17626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:43.204275 17626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:43.204380 17626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:43.204450 17626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:43.215772 17626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:42457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:42457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:43.217168 17626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:43.218463 17626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:43.230044 17631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.230124 17634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.230157 17632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.231323 17626 server_base.cc:1034] running on GCE node
I20260927 18:49:43.232175 17626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:43.233762 17626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:43.235014 17626 hybrid_clock.cc:648] HybridClock initialized: now 1790534983234959 us; error 64 us; skew 500 ppm
I20260927 18:49:43.235407 17626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:43 dist-test-slave-rfd5 krb5kdc[17613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:43.238794 17626 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:49:43.240360 17626 webserver.cc:460] Webserver started at http://127.31.26.254:38595/ using document root <none> and password file <none>
I20260927 18:49:43.241129 17626 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:43.241284 17626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:43.241608 17626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:43.244696 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "f413b5d47008495e84a4adc27b5d6431"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-rfd5"
server_key: "ba68127ebe9cdcd237e9da34905d73c0"
server_key_iv: "2be6ac36336ecb16de46f8d75c09ab5e"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.245548 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "f413b5d47008495e84a4adc27b5d6431"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-rfd5"
server_key: "ba68127ebe9cdcd237e9da34905d73c0"
server_key_iv: "2be6ac36336ecb16de46f8d75c09ab5e"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.251324 17626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:49:43.255529 17641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.257218 17626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:43.257442 17626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "f413b5d47008495e84a4adc27b5d6431"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-rfd5"
server_key: "ba68127ebe9cdcd237e9da34905d73c0"
server_key_iv: "2be6ac36336ecb16de46f8d75c09ab5e"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.257781 17626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:43.271425 17626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:43.276398 17626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:43.276706 17626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:43.296075 17626 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:42457
I20260927 18:49:43.296098 17692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:42457 every 8 connection(s)
I20260927 18:49:43.297600 17626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:43.298699 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17626
I20260927 18:49:43.299054 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:43.299532 31851 external_mini_cluster.cc:1428] Setting key 9042385494b6f6f81dc3f01eba7759ea
I20260927 18:49:43.302146 17693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:49:43 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:43.316128 17693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431: Bootstrap starting.
I20260927 18:49:43.320888 17693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:43.322247 17693 log.cc:824] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:43.325958 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.301482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.31.26.254:42457)
0927 18:49:43.302290 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:43.302307 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:43.302362 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:49:43.304008 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:43.304030 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:43.304409 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:43.304760 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:43.306810 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.308179 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.310178 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.310942 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.311099 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.316293 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:43.316307 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:43.316331 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:43.316383 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:43.320622 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.321668 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.321677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.321687 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.321925 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.322351 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.322356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.322359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.323153 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:43.323388 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:43.324056 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 18:49:43.324506 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":378,"threads_started":1}
I20260927 18:49:43.326066 17693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431: No bootstrap required, opened a new log
I20260927 18:49:43.334617 17693 raft_consensus.cc:348] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f413b5d47008495e84a4adc27b5d6431" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42457 } }
I20260927 18:49:43.335165 17693 raft_consensus.cc:374] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:43.335304 17693 raft_consensus.cc:729] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f413b5d47008495e84a4adc27b5d6431, State: Initialized, Role: FOLLOWER
I20260927 18:49:43.336169 17693 consensus_queue.cc:260] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f413b5d47008495e84a4adc27b5d6431" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42457 } }
I20260927 18:49:43.336459 17693 raft_consensus.cc:388] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:43.336609 17693 raft_consensus.cc:482] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:43.336807 17693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:43.339188 17693 raft_consensus.cc:504] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f413b5d47008495e84a4adc27b5d6431" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42457 } }
I20260927 18:49:43.339845 17693 leader_election.cc:302] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f413b5d47008495e84a4adc27b5d6431; no voters:
I20260927 18:49:43.340581 17693 leader_election.cc:288] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:43.340788 17698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:43.341751 17698 raft_consensus.cc:686] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [term 1 LEADER]: Becoming Leader. State: Replica: f413b5d47008495e84a4adc27b5d6431, State: Running, Role: LEADER
I20260927 18:49:43.342357 17698 consensus_queue.cc:237] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f413b5d47008495e84a4adc27b5d6431" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42457 } }
I20260927 18:49:43.343494 17693 sys_catalog.cc:564] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:43.345284 17700 sys_catalog.cc:455] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f413b5d47008495e84a4adc27b5d6431. Latest consensus state: current_term: 1 leader_uuid: "f413b5d47008495e84a4adc27b5d6431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f413b5d47008495e84a4adc27b5d6431" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42457 } } }
I20260927 18:49:43.345314 17699 sys_catalog.cc:455] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f413b5d47008495e84a4adc27b5d6431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f413b5d47008495e84a4adc27b5d6431" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 42457 } } }
I20260927 18:49:43.345768 17700 sys_catalog.cc:458] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:43.345763 17699 sys_catalog.cc:458] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:43.348037 17703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:43.353987 17703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:43.365633 17703 catalog_manager.cc:1347] Generated new cluster ID: f587949d453e426f81eaa284ffd934cf
I20260927 18:49:43.365764 17703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:43.375221 17703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:43.376796 17703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:43.387136 17703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431: Generated new TSK 0
I20260927 18:49:43.388182 17703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:49:43.458321 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42457
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:43.808712 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:43.809029 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:43.809214 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:43.816911 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:43.817046 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:43.817276 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:43.827534 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:43.828999 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:43.830405 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:43.844300 17724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.844252 17727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.844323 17725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.845361 17719 server_base.cc:1034] running on GCE node
I20260927 18:49:43.846128 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:43.847451 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:43.848702 17719 hybrid_clock.cc:648] HybridClock initialized: now 1790534983848658 us; error 62 us; skew 500 ppm
I20260927 18:49:43.849056 17719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:43 dist-test-slave-rfd5 krb5kdc[17613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:43.852406 17719 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:49:43.854061 17719 webserver.cc:460] Webserver started at http://127.31.26.193:43043/ using document root <none> and password file <none>
I20260927 18:49:43.854760 17719 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:43.854864 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:43.855152 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:43.858266 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "b8da4037a44e4583800ce10ba6a189fd"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-rfd5"
server_key: "5ca2cdd7d00f80e79b1ada30a51c15b2"
server_key_iv: "262f104de6af56baf5db4e4e062a4bb2"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.859031 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "b8da4037a44e4583800ce10ba6a189fd"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-rfd5"
server_key: "5ca2cdd7d00f80e79b1ada30a51c15b2"
server_key_iv: "262f104de6af56baf5db4e4e062a4bb2"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.864480 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:49:43.868121 17734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.869794 17719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:49:43.869948 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "b8da4037a44e4583800ce10ba6a189fd"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-rfd5"
server_key: "5ca2cdd7d00f80e79b1ada30a51c15b2"
server_key_iv: "262f104de6af56baf5db4e4e062a4bb2"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.870150 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:43.895267 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:43.900228 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:43.900740 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:43.901913 17719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:43.903285 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:43.903358 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.903523 17719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:43.903592 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.937762 17719 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:34333
I20260927 18:49:43.937857 17846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:34333 every 8 connection(s)
I20260927 18:49:43.939164 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:43.947655 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17719
I20260927 18:49:43.947851 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:43.948473 31851 external_mini_cluster.cc:1428] Setting key 7688e7fdfa25aacdb130f01a8f363f98
Sep 27 18:49:43 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:43.959820 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.941852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:53787 (local address 127.31.26.254:42457)
0927 18:49:43.942331 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:43.942336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:43.943235 (+ 899us) server_negotiation.cc:407] Connection header received
0927 18:49:43.945282 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:43.945288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:43.945382 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:43.945501 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:43.947000 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.947781 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.949532 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.949961 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.950120 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.953903 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:43.953908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:43.953911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:43.953946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:43.956578 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.957442 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.957446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.957449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.957572 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.958022 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.958026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.958028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.958300 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:43.958462 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:43.959213 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 18:49:43.959449 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":263,"threads_started":1}
I20260927 18:49:43.960860 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.942339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42457 (local address 127.31.26.193:53787)
0927 18:49:43.943069 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:43.943107 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:43.944961 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:43.945750 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:43.945758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:43.946265 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:43.946823 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:43.946840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.947957 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.947964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:43.949388 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:43.949400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.950211 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.950220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:43.950475 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.951190 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:43.951212 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:43.953671 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:43.956741 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:43.956750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:43.956770 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:43.957302 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:43.957687 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:43.957693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:43.957697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:43.957891 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:43.958441 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:43.958449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:43.958816 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 18:49:43.959107 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 18:49:43.959692 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":282,"threads_started":1}
I20260927 18:49:43.962719 17847 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42457
I20260927 18:49:43.963099 17847 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:43.964010 17847 heartbeater.cc:511] Master 127.31.26.254:42457 requested a full tablet report, sending...
I20260927 18:49:43.966636 17658 ts_manager.cc:194] Registered new tserver with Master: b8da4037a44e4583800ce10ba6a189fd (127.31.26.193:34333)
I20260927 18:49:43.968907 17658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:53787
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:49:44.004814 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42457
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:44.348824 17853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:44.349347 17853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:44.349524 17853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:44.357405 17853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:44.357586 17853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:44.357820 17853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:44.369669 17853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:44.371060 17853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:44.372406 17853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:44.386701 17858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:44.386737 17859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:44.386647 17861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:44.388289 17853 server_base.cc:1034] running on GCE node
I20260927 18:49:44.389156 17853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:44.390309 17853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:44.391593 17853 hybrid_clock.cc:648] HybridClock initialized: now 1790534984391518 us; error 70 us; skew 500 ppm
I20260927 18:49:44.391980 17853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:44 dist-test-slave-rfd5 krb5kdc[17613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534984, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:44.395529 17853 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:49:44.397272 17853 webserver.cc:460] Webserver started at http://127.31.26.194:36227/ using document root <none> and password file <none>
I20260927 18:49:44.398039 17853 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:44.398198 17853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:44.398541 17853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:44.402061 17853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "f383cc492c3f4f53ab22e012d38a84a6"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-rfd5"
server_key: "ab5b46c3ba76b83fbb874971ba0dd639"
server_key_iv: "057ce029d5837c14f1f02dbf61942958"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.402891 17853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "f383cc492c3f4f53ab22e012d38a84a6"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-rfd5"
server_key: "ab5b46c3ba76b83fbb874971ba0dd639"
server_key_iv: "057ce029d5837c14f1f02dbf61942958"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.408365 17853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:49:44.412317 17868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.414005 17853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:44.414207 17853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "f383cc492c3f4f53ab22e012d38a84a6"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-rfd5"
server_key: "ab5b46c3ba76b83fbb874971ba0dd639"
server_key_iv: "057ce029d5837c14f1f02dbf61942958"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.414431 17853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:44.445865 17853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:44.450440 17853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:44.450768 17853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:44.451817 17853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:44.453277 17853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:44.453392 17853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.453559 17853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:44.453644 17853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.489645 17853 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:42779
I20260927 18:49:44.489713 17980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:42779 every 8 connection(s)
I20260927 18:49:44.491178 17853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:44.501008 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17853
I20260927 18:49:44.501274 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:44.501845 31851 external_mini_cluster.cc:1428] Setting key 81716ce9905c921591ad635b9027fc13
Sep 27 18:49:44 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534984, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:44.515494 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.494190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:56869 (local address 127.31.26.254:42457)
0927 18:49:44.494730 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.494737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.495900 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 18:49:44.497958 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.497964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.498047 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.498166 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:44.500001 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.500783 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.503328 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.504110 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.504257 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.509039 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:44.509045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:44.509048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:44.509140 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:44.512027 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:44.512915 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:44.512919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:44.512922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:44.513066 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:44.513594 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:44.513599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:44.513603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:44.513889 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:44.514057 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.514849 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.515104 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":258,"threads_started":1}
I20260927 18:49:44.516593 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.494881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42457 (local address 127.31.26.194:56869)
0927 18:49:44.495704 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.495747 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.497632 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.498470 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.498480 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.499101 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:44.499753 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.499773 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.500996 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.501002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.503140 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.503156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.504403 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.504412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.504773 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.505633 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:44.505671 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:44.508707 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:44.512216 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:44.512226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:44.512246 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:44.512754 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:44.513244 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:44.513249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:44.513253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:44.513461 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:44.514022 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:44.514031 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:44.514460 (+ 429us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.514761 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.515336 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
I20260927 18:49:44.518402 17981 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42457
I20260927 18:49:44.518770 17981 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:44.519757 17981 heartbeater.cc:511] Master 127.31.26.254:42457 requested a full tablet report, sending...
I20260927 18:49:44.521607 17658 ts_manager.cc:194] Registered new tserver with Master: f383cc492c3f4f53ab22e012d38a84a6 (127.31.26.194:42779)
I20260927 18:49:44.522929 17658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:56869
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:49:44.561492 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42457
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:44.904172 17987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:44.904476 17987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:44.904572 17987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:44.911949 17987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:44.912088 17987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:44.912205 17987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:44.922493 17987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:42457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:44.923802 17987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:44.925077 17987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:44.938933 17992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:44.938889 17993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:44.939786 17995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:44.940502 17987 server_base.cc:1034] running on GCE node
I20260927 18:49:44.941217 17987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:44.942293 17987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:44.943533 17987 hybrid_clock.cc:648] HybridClock initialized: now 1790534984943482 us; error 49 us; skew 500 ppm
I20260927 18:49:44.943877 17987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:44 dist-test-slave-rfd5 krb5kdc[17613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534984, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:44.947286 17987 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:49:44.948995 17987 webserver.cc:460] Webserver started at http://127.31.26.195:42733/ using document root <none> and password file <none>
I20260927 18:49:44.949831 17987 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:44.949980 17987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:44.950295 17987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:44.953392 17987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "966469e9c5e3451b9404c811f3ced831"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-rfd5"
server_key: "9a9f646195b083139ccda65446bf9074"
server_key_iv: "8ea0da7417c56f34748ce242bccd040f"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.954206 17987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "966469e9c5e3451b9404c811f3ced831"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-rfd5"
server_key: "9a9f646195b083139ccda65446bf9074"
server_key_iv: "8ea0da7417c56f34748ce242bccd040f"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.961263 17987 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 18:49:44.965750 18002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.967458 17987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 18:49:44.967657 17987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "966469e9c5e3451b9404c811f3ced831"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-rfd5"
server_key: "9a9f646195b083139ccda65446bf9074"
server_key_iv: "8ea0da7417c56f34748ce242bccd040f"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.967869 17987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:44.972283 17847 heartbeater.cc:503] Master 127.31.26.254:42457 was elected leader, sending a full tablet report...
I20260927 18:49:44.995622 17987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:45.000164 17987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:45.000476 17987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:45.001559 17987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:45.003039 17987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:45.003168 17987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:45.003347 17987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:45.003420 17987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:45.037392 17987 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35561
I20260927 18:49:45.037441 18114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35561 every 8 connection(s)
I20260927 18:49:45.038780 17987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:45.048662 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 17987
I20260927 18:49:45.048868 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:45.049412 31851 external_mini_cluster.cc:1428] Setting key b0b54e4bbf9aa939b6e78c7e6c95ba5e
Sep 27 18:49:45 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534984, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:45.059675 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.041380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:56957 (local address 127.31.26.254:42457)
0927 18:49:45.041820 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.041825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.042886 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 18:49:45.044892 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.044898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.045007 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.045150 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:45.046677 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.047466 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.049412 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.049913 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.050063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.053937 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:45.053942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:45.053946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:45.053980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:45.056793 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.057623 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.057628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.057630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.057761 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.058142 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.058145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.058147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.058414 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:45.058529 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.059197 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.059373 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260927 18:49:45.060731 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.041997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:42457 (local address 127.31.26.195:56957)
0927 18:49:45.042723 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.042760 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:45.044575 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:45.045391 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:45.045400 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:45.045916 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:45.046503 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.046520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.047621 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.047629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.049256 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.049284 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.050171 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.050178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.050452 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.051258 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:45.051288 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:45.053737 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:45.056931 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:45.056938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:45.056955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:45.057488 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:45.057853 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:45.057856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:45.057858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:45.058041 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:45.058530 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:45.058536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:45.058863 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 18:49:45.059106 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 18:49:45.059516 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":275,"threads_started":1}
I20260927 18:49:45.062232 18115 heartbeater.cc:347] Connected to a master server at 127.31.26.254:42457
I20260927 18:49:45.062572 18115 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:45.063534 18115 heartbeater.cc:511] Master 127.31.26.254:42457 requested a full tablet report, sending...
I20260927 18:49:45.065330 17658 ts_manager.cc:194] Registered new tserver with Master: 966469e9c5e3451b9404c811f3ced831 (127.31.26.195:35561)
I20260927 18:49:45.066141 17658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:56957
I20260927 18:49:45.077432 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:45.095191 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.081635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.31.26.254:42457)
0927 18:49:45.081862 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.081871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.082257 (+ 386us) server_negotiation.cc:407] Connection header received
0927 18:49:45.082471 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.082475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.082542 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.082655 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:45.084063 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.084821 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.086259 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.086672 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.086813 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.089651 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:45.089657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:45.089660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:45.089694 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:45.092075 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.092938 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.092944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.092948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.093130 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.093625 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.093631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.093634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.093950 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:45.094223 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.094667 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.094869 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:49:45.099237 17658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:45.106382 17658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:45.145126 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.118968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44540)
0927 18:49:45.130532 (+ 11564us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.130591 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:45.130813 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:45.137337 (+ 6524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:45.137348 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:45.137396 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:45.138075 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.138089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.139880 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.139889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.142465 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.142478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.143886 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.143892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.144085 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.144141 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 18:49:45.144302 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:49:45.144506 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11275,"mutex_wait_us":154,"spinlock_wait_cycles":7040,"thread_start_us":12638,"threads_started":1}
I20260927 18:49:45.144898 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.119289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.31.26.193:34333)
0927 18:49:45.136517 (+ 17228us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.136527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.136558 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:49:45.136652 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.136659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.136856 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.137074 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:45.138245 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.139730 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.142627 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.143680 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.143838 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.144151 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.144359 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.144527 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17048,"thread_start_us":16890,"threads_started":1}
I20260927 18:49:45.145798 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.119610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:42779 (local address 127.0.0.1:37766)
0927 18:49:45.135681 (+ 16071us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.135705 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:45.135854 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:45.136934 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:45.136945 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:45.136982 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:45.137576 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.137595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.139606 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.139611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.142520 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.142542 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.144773 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.144779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.145038 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.145077 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:49:45.145183 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:49:45.145330 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15908,"mutex_wait_us":197,"thread_start_us":15721,"threads_started":1}
I20260927 18:49:45.146392 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.119903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.31.26.195:35561)
0927 18:49:45.136213 (+ 16310us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.136223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.136258 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:49:45.136352 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.136359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.136625 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.136823 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:45.138235 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.139262 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.142710 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.144252 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.144432 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.144626 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.145662 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.146046 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16078,"thread_start_us":15862,"threads_started":1}
I20260927 18:49:45.146644 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.120139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.31.26.194:42779)
0927 18:49:45.135113 (+ 14974us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.135127 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.135985 (+ 858us) server_negotiation.cc:407] Connection header received
0927 18:49:45.136115 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.136126 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.136427 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.136654 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:45.137795 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.139466 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.142715 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.144252 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.144383 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.144592 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.146102 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.146282 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14746,"thread_start_us":14576,"threads_started":1}
I20260927 18:49:45.147275 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.119176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35561 (local address 127.0.0.1:41874)
0927 18:49:45.131945 (+ 12769us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.131972 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:45.132136 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:45.137178 (+ 5042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:45.137182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:45.137233 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:45.137967 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.137984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.139408 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.139416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.142520 (+ 3104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.142535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.144832 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.144837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.145069 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.145124 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 18:49:45.145183 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 18:49:45.146908 (+ 1725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12595,"mutex_wait_us":269,"spinlock_wait_cycles":1408,"thread_start_us":10922,"threads_started":1}
I20260927 18:49:45.149348 17782 tablet_service.cc:1467] Processing CreateTablet for tablet 1160c18816bd41e88fe90ca3682a0f7d (DEFAULT_TABLE table=test-table [id=d8ff9ef3e7c342d7bc52e82b7875ffff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:45.150106 17916 tablet_service.cc:1467] Processing CreateTablet for tablet 1160c18816bd41e88fe90ca3682a0f7d (DEFAULT_TABLE table=test-table [id=d8ff9ef3e7c342d7bc52e82b7875ffff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:45.151005 17782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1160c18816bd41e88fe90ca3682a0f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:45.151479 17916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1160c18816bd41e88fe90ca3682a0f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:45.152549 18050 tablet_service.cc:1467] Processing CreateTablet for tablet 1160c18816bd41e88fe90ca3682a0f7d (DEFAULT_TABLE table=test-table [id=d8ff9ef3e7c342d7bc52e82b7875ffff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:45.154030 18050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1160c18816bd41e88fe90ca3682a0f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:45.162637 18134 tablet_bootstrap.cc:492] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: Bootstrap starting.
I20260927 18:49:45.166366 18134 tablet_bootstrap.cc:654] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:45.167457 18134 log.cc:824] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:45.170342 18134 tablet_bootstrap.cc:492] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: No bootstrap required, opened a new log
I20260927 18:49:45.170620 18134 ts_tablet_manager.cc:1397] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:49:45.175324 18137 tablet_bootstrap.cc:492] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Bootstrap starting.
I20260927 18:49:45.178426 18137 tablet_bootstrap.cc:654] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:45.178825 18135 tablet_bootstrap.cc:492] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Bootstrap starting.
I20260927 18:49:45.178674 18134 raft_consensus.cc:348] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.179155 18134 raft_consensus.cc:374] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:45.179263 18134 raft_consensus.cc:729] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8da4037a44e4583800ce10ba6a189fd, State: Initialized, Role: FOLLOWER
I20260927 18:49:45.179896 18137 log.cc:824] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:45.180050 18134 consensus_queue.cc:260] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.181983 18134 ts_tablet_manager.cc:1428] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260927 18:49:45.183326 18135 tablet_bootstrap.cc:654] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:45.184659 18135 log.cc:824] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:45.184732 18137 tablet_bootstrap.cc:492] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: No bootstrap required, opened a new log
I20260927 18:49:45.185003 18137 ts_tablet_manager.cc:1397] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260927 18:49:45.188063 18135 tablet_bootstrap.cc:492] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: No bootstrap required, opened a new log
I20260927 18:49:45.188330 18135 ts_tablet_manager.cc:1397] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
W20260927 18:49:45.191468 17848 tablet.cc:2369] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:45.195909 18137 raft_consensus.cc:348] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.196400 18137 raft_consensus.cc:374] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:45.196507 18137 raft_consensus.cc:729] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966469e9c5e3451b9404c811f3ced831, State: Initialized, Role: FOLLOWER
I20260927 18:49:45.197206 18137 consensus_queue.cc:260] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.198367 18115 heartbeater.cc:503] Master 127.31.26.254:42457 was elected leader, sending a full tablet report...
I20260927 18:49:45.199155 18137 ts_tablet_manager.cc:1428] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260927 18:49:45.198833 18135 raft_consensus.cc:348] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.199304 18135 raft_consensus.cc:374] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:45.199411 18135 raft_consensus.cc:729] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f383cc492c3f4f53ab22e012d38a84a6, State: Initialized, Role: FOLLOWER
I20260927 18:49:45.201025 18135 consensus_queue.cc:260] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.202236 17981 heartbeater.cc:503] Master 127.31.26.254:42457 was elected leader, sending a full tablet report...
I20260927 18:49:45.202708 18135 ts_tablet_manager.cc:1428] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260927 18:49:45.243597 17982 tablet.cc:2369] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:45.275687 18142 raft_consensus.cc:482] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:45.276127 18142 raft_consensus.cc:504] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.278007 18142 leader_election.cc:288] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8da4037a44e4583800ce10ba6a189fd (127.31.26.193:34333), 966469e9c5e3451b9404c811f3ced831 (127.31.26.195:35561)
I20260927 18:49:45.285735 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.278797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:59095 (local address 127.31.26.195:35561)
0927 18:49:45.279018 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.279026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.279813 (+ 787us) server_negotiation.cc:407] Connection header received
0927 18:49:45.279962 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.279996 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.280085 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.280167 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:45.280852 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.281833 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.283862 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.284640 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.284834 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.284921 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.285333 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.285464 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:49:45.285758 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.278486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:53079 (local address 127.31.26.193:34333)
0927 18:49:45.278725 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.278731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.279411 (+ 680us) server_negotiation.cc:407] Connection header received
0927 18:49:45.279721 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.279725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.279820 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.279917 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:45.280950 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.281848 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.283848 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.284642 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.284840 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.284926 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.285359 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.285493 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:49:45.285938 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.278530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35561 (local address 127.31.26.194:59095)
0927 18:49:45.279701 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.279719 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:45.279834 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:45.280198 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:45.280202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:45.280222 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:45.280674 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.280685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.281964 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.281972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.283687 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.283700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.284889 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.284895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.285150 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.285191 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:49:45.285299 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:49:45.285444 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":65,"thread_start_us":827,"threads_started":1}
I20260927 18:49:45.286330 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.278211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.31.26.194:53079)
0927 18:49:45.279279 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.279296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:45.279544 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:45.280324 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:45.280330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:45.280376 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:45.280802 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.280811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.281987 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.281993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.283692 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:45.283704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.284922 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.284929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:45.285191 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.285216 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:49:45.285320 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:49:45.285450 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":62,"thread_start_us":728,"threads_started":1}
I20260927 18:49:45.287024 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160c18816bd41e88fe90ca3682a0f7d" candidate_uuid: "f383cc492c3f4f53ab22e012d38a84a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "966469e9c5e3451b9404c811f3ced831" is_pre_election: true
I20260927 18:49:45.287375 17802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160c18816bd41e88fe90ca3682a0f7d" candidate_uuid: "f383cc492c3f4f53ab22e012d38a84a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8da4037a44e4583800ce10ba6a189fd" is_pre_election: true
I20260927 18:49:45.287649 18070 raft_consensus.cc:2454] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f383cc492c3f4f53ab22e012d38a84a6 in term 0.
I20260927 18:49:45.287930 17802 raft_consensus.cc:2454] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f383cc492c3f4f53ab22e012d38a84a6 in term 0.
I20260927 18:49:45.288436 17871 leader_election.cc:302] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 966469e9c5e3451b9404c811f3ced831, f383cc492c3f4f53ab22e012d38a84a6; no voters:
I20260927 18:49:45.288869 18142 raft_consensus.cc:2781] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:45.289018 18142 raft_consensus.cc:482] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:45.289144 18142 raft_consensus.cc:3037] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:49:45.290416 18116 tablet.cc:2369] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:45.291783 18142 raft_consensus.cc:504] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.292492 18142 leader_election.cc:288] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [CANDIDATE]: Term 1 election: Requested vote from peers b8da4037a44e4583800ce10ba6a189fd (127.31.26.193:34333), 966469e9c5e3451b9404c811f3ced831 (127.31.26.195:35561)
I20260927 18:49:45.292970 17802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160c18816bd41e88fe90ca3682a0f7d" candidate_uuid: "f383cc492c3f4f53ab22e012d38a84a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8da4037a44e4583800ce10ba6a189fd"
I20260927 18:49:45.293169 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160c18816bd41e88fe90ca3682a0f7d" candidate_uuid: "f383cc492c3f4f53ab22e012d38a84a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "966469e9c5e3451b9404c811f3ced831"
I20260927 18:49:45.293251 17802 raft_consensus.cc:3037] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:45.293394 18070 raft_consensus.cc:3037] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:45.296648 18070 raft_consensus.cc:2454] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f383cc492c3f4f53ab22e012d38a84a6 in term 1.
I20260927 18:49:45.296648 17802 raft_consensus.cc:2454] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f383cc492c3f4f53ab22e012d38a84a6 in term 1.
I20260927 18:49:45.297067 17871 leader_election.cc:302] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 966469e9c5e3451b9404c811f3ced831, f383cc492c3f4f53ab22e012d38a84a6; no voters:
I20260927 18:49:45.297518 18142 raft_consensus.cc:2781] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:45.298000 18142 raft_consensus.cc:686] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 1 LEADER]: Becoming Leader. State: Replica: f383cc492c3f4f53ab22e012d38a84a6, State: Running, Role: LEADER
I20260927 18:49:45.298566 18142 consensus_queue.cc:237] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } }
I20260927 18:49:45.303248 17657 catalog_manager.cc:5495] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f383cc492c3f4f53ab22e012d38a84a6 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "f383cc492c3f4f53ab22e012d38a84a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f383cc492c3f4f53ab22e012d38a84a6" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42779 } health_report { overall_health: HEALTHY } } }
I20260927 18:49:45.340240 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.335146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.31.26.194:42779)
0927 18:49:45.335341 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.335347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.335365 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:49:45.335527 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.335531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.335615 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.335741 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:45.336094 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.336789 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.338238 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.338635 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.338739 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.339200 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:45.339697 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:45.339830 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.339966 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.340006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:49:45.348940 18070 raft_consensus.cc:1261] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 1 FOLLOWER]: Refusing update from remote peer f383cc492c3f4f53ab22e012d38a84a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:45.348940 17802 raft_consensus.cc:1261] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 1 FOLLOWER]: Refusing update from remote peer f383cc492c3f4f53ab22e012d38a84a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:45.350133 18145 consensus_queue.cc:1035] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "966469e9c5e3451b9404c811f3ced831" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:45.350523 18142 consensus_queue.cc:1035] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8da4037a44e4583800ce10ba6a189fd" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 34333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:45.363346 18148 mvcc.cc:205] Tried to move back new op lower bound from 7334031299978387456 to 7334031299787046912. Current Snapshot: MvccSnapshot[applied={T|T < 7334031299978387456}]
I20260927 18:49:45.377031 17658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:45.377418 17658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:45.381435 17658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f413b5d47008495e84a4adc27b5d6431: Sending DeleteTablet for 3 replicas of tablet 1160c18816bd41e88fe90ca3682a0f7d
I20260927 18:49:45.382671 17782 tablet_service.cc:1514] Processing DeleteTablet for tablet 1160c18816bd41e88fe90ca3682a0f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:45 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:44540
I20260927 18:49:45.383121 17916 tablet_service.cc:1514] Processing DeleteTablet for tablet 1160c18816bd41e88fe90ca3682a0f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:45 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:37766
I20260927 18:49:45.384066 18050 tablet_service.cc:1514] Processing DeleteTablet for tablet 1160c18816bd41e88fe90ca3682a0f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:45 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:41874
I20260927 18:49:45.384959 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17719
I20260927 18:49:45.407757 18160 tablet_replica.cc:330] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: stopping tablet replica
I20260927 18:49:45.408447 18159 tablet_replica.cc:330] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: stopping tablet replica
I20260927 18:49:45.409009 18160 raft_consensus.cc:2229] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:45.409603 18159 raft_consensus.cc:2229] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:45.410203 18160 raft_consensus.cc:2258] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:45.410938 18159 raft_consensus.cc:2258] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:45.413545 18160 ts_tablet_manager.cc:1905] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:45.415575 18159 ts_tablet_manager.cc:1905] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:49:45 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:49:45.430502 18160 ts_tablet_manager.cc:1918] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:45.430979 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.386234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.31.26.193:34333)
0927 18:49:45.386495 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.386503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.402351 (+ 15848us) server_negotiation.cc:407] Connection header received
0927 18:49:45.403006 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.403016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.403135 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.403232 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:45.406297 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.407931 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.411173 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.411955 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.412069 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.419940 (+ 7871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:45.419970 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:45.419997 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:45.420072 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:45.425572 (+ 5500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.426984 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.426999 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.427005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.427263 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.427908 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.427916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.427922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.428607 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:45.429004 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.429864 (+ 860us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.430212 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:49:45.431336 18158 tablet_replica.cc:330] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd: stopping tablet replica
I20260927 18:49:45.431831 18158 raft_consensus.cc:2229] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:45.432516 17842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:45.432482 18158 raft_consensus.cc:2258] T 1160c18816bd41e88fe90ca3682a0f7d P b8da4037a44e4583800ce10ba6a189fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:45.433415 18159 ts_tablet_manager.cc:1918] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:45.433635 18159 log.cc:1196] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/1160c18816bd41e88fe90ca3682a0f7d
I20260927 18:49:45.434064 18160 log.cc:1196] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/1160c18816bd41e88fe90ca3682a0f7d
I20260927 18:49:45.434922 18160 ts_tablet_manager.cc:1939] T 1160c18816bd41e88fe90ca3682a0f7d P 966469e9c5e3451b9404c811f3ced831: Deleting consensus metadata
I20260927 18:49:45.437275 18159 ts_tablet_manager.cc:1939] T 1160c18816bd41e88fe90ca3682a0f7d P f383cc492c3f4f53ab22e012d38a84a6: Deleting consensus metadata
I20260927 18:49:45.437261 17644 catalog_manager.cc:4841] TS 966469e9c5e3451b9404c811f3ced831 (127.31.26.195:35561): tablet 1160c18816bd41e88fe90ca3682a0f7d (table test-table [id=d8ff9ef3e7c342d7bc52e82b7875ffff]) successfully deleted
I20260927 18:49:45.439081 17645 catalog_manager.cc:4841] TS f383cc492c3f4f53ab22e012d38a84a6 (127.31.26.194:42779): tablet 1160c18816bd41e88fe90ca3682a0f7d (table test-table [id=d8ff9ef3e7c342d7bc52e82b7875ffff]) successfully deleted
I20260927 18:49:45.543072 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17719
W20260927 18:49:45.561754 17643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.26.193:34333 (error 108)
I20260927 18:49:45.562543 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17853
I20260927 18:49:45.564064 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.563541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44566)
0927 18:49:45.563741 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.563880 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 18:49:45.564513 17643 catalog_manager.cc:4584] TS b8da4037a44e4583800ce10ba6a189fd (127.31.26.193:34333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1160c18816bd41e88fe90ca3682a0f7d: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Sep 27 18:49:45 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:49:45.580873 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.563423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.31.26.194:42779)
0927 18:49:45.563837 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.563843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.563866 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:49:45.563945 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.563951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.564055 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.564177 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:45.565720 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.566970 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.568274 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.568994 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.569176 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.573937 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:45.573956 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:45.573977 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:45.574051 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:45.577944 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.578711 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.578721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.578724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.578891 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.579254 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.579259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.579262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.579755 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:45.580101 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.580249 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.580475 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260927 18:49:45.581866 17976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:45.618304 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.617707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44580)
0927 18:49:45.617961 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.618076 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 18:49:45.619494 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.619077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44582)
0927 18:49:45.619250 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.619355 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:49:45.680284 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.679725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44590)
0927 18:49:45.679979 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.680097 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 18:49:45.681501 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.680990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44604)
0927 18:49:45.681259 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.681346 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:49:45.701071 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17853
I20260927 18:49:45.720165 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17987
Sep 27 18:49:45 dist-test-slave-rfd5 krb5kdc[17613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:49:45.735450 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.720963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.31.26.195:35561)
0927 18:49:45.721191 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:45.721195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:45.721220 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:49:45.721332 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:45.721336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:45.721408 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:45.721495 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:45.722866 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.723561 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.724758 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:45.725227 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:45.725321 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:45.729945 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:45.729960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:45.730002 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:45.730039 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:45.732721 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.733507 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.733515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.733517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.733660 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:45.733984 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:45.733987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:45.733989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:45.734336 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:45.734551 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:45.734847 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 18:49:45.735127 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:49:45.736079 18110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:45.761899 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.761433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44610)
0927 18:49:45.761624 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.761723 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:49:45.763027 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.762640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44612)
0927 18:49:45.762846 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.762916 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 18:49:45.827926 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17987
I20260927 18:49:45.844717 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 17626
I20260927 18:49:45.845950 17688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:45.959704 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:45.959055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:34333 (local address 127.0.0.1:44622)
0927 18:49:45.959278 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:45.959394 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 18:49:45.959827 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 17626
2026-09-27T18:49:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4565 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T18:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:46Z Disabled control of system clock
I20260927 18:49:46.036167 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:46875
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:46875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:49:46.389497 18170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:46.389945 18170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:46.390031 18170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:46.397374 18170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:46.397498 18170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:46.397536 18170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:46.397596 18170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:46.397641 18170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:46.407289 18170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:46875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:46.408586 18170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:46.410466 18170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:46.423058 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:46.423087 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:46.423122 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:46.424222 18170 server_base.cc:1034] running on GCE node
I20260927 18:49:46.425252 18170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:46.426822 18170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:46.428041 18170 hybrid_clock.cc:648] HybridClock initialized: now 1790534986427978 us; error 52 us; skew 500 ppm
I20260927 18:49:46.428458 18170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:46.436775 18170 webserver.cc:460] Webserver started at http://127.31.26.254:42831/ using document root <none> and password file <none>
I20260927 18:49:46.437848 18170 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:46.437978 18170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:46.438361 18170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:46.441785 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "7e0ca64961d945b4aea4771e8eeed908"
format_stamp: "Formatted at 2026-09-27 18:49:46 on dist-test-slave-rfd5"
server_key: "053c58573da5cab98abb31a7f53c7d52"
server_key_iv: "c3e376bf9c5384dcd886524ec0f32f55"
server_key_version: "encryptionkey@0"
I20260927 18:49:46.442683 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "7e0ca64961d945b4aea4771e8eeed908"
format_stamp: "Formatted at 2026-09-27 18:49:46 on dist-test-slave-rfd5"
server_key: "053c58573da5cab98abb31a7f53c7d52"
server_key_iv: "c3e376bf9c5384dcd886524ec0f32f55"
server_key_version: "encryptionkey@0"
I20260927 18:49:46.448477 18170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:49:46.452598 18184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:46.454190 18170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:46.454409 18170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "7e0ca64961d945b4aea4771e8eeed908"
format_stamp: "Formatted at 2026-09-27 18:49:46 on dist-test-slave-rfd5"
server_key: "053c58573da5cab98abb31a7f53c7d52"
server_key_iv: "c3e376bf9c5384dcd886524ec0f32f55"
server_key_version: "encryptionkey@0"
I20260927 18:49:46.454696 18170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:46.508836 18170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:46.509871 18170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:46.510178 18170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:46.530159 18170 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:46875
I20260927 18:49:46.530184 18235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:46875 every 8 connection(s)
I20260927 18:49:46.531633 18170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:46.532647 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18170
I20260927 18:49:46.532850 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:46.533406 31851 external_mini_cluster.cc:1428] Setting key 2f16727d178fe093a0911b8ddf165778
I20260927 18:49:46.536664 18236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:46.547693 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:46.535456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.31.26.254:46875)
0927 18:49:46.536478 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:46.536497 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:46.536540 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:49:46.538369 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:46.538393 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:46.538882 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:46.539329 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:46.539949 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:46.541376 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:46.542888 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:46.543595 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:46.543806 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:46.545130 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:46.545142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:46.545165 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:46.545807 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:46.545912 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:46.545939 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:46.546091 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:46.546324 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 18:49:46.546677 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":350,"threads_started":1}
I20260927 18:49:46.548126 18236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908: Bootstrap starting.
I20260927 18:49:46.551663 18236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:46.552724 18236 log.cc:824] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:46.555850 18236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908: No bootstrap required, opened a new log
I20260927 18:49:46.564196 18236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ca64961d945b4aea4771e8eeed908" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 46875 } }
I20260927 18:49:46.564632 18236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:46.564738 18236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0ca64961d945b4aea4771e8eeed908, State: Initialized, Role: FOLLOWER
I20260927 18:49:46.565461 18236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ca64961d945b4aea4771e8eeed908" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 46875 } }
I20260927 18:49:46.565698 18236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:46.565841 18236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:46.566037 18236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:46.568413 18236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ca64961d945b4aea4771e8eeed908" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 46875 } }
I20260927 18:49:46.568954 18236 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0ca64961d945b4aea4771e8eeed908; no voters:
I20260927 18:49:46.569613 18236 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:46.569788 18241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:46.570502 18241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0ca64961d945b4aea4771e8eeed908, State: Running, Role: LEADER
I20260927 18:49:46.571103 18241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ca64961d945b4aea4771e8eeed908" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 46875 } }
I20260927 18:49:46.571785 18236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:46.574568 18243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0ca64961d945b4aea4771e8eeed908. Latest consensus state: current_term: 1 leader_uuid: "7e0ca64961d945b4aea4771e8eeed908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ca64961d945b4aea4771e8eeed908" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 46875 } } }
I20260927 18:49:46.575090 18243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:46.574697 18242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0ca64961d945b4aea4771e8eeed908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ca64961d945b4aea4771e8eeed908" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 46875 } } }
I20260927 18:49:46.575253 18242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:46.576210 18247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:46.582188 18247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:46.591511 18247 catalog_manager.cc:1347] Generated new cluster ID: 020e080fa2864f76bc7edab67c16d8d7
I20260927 18:49:46.591648 18247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:46.623708 18247 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:46.625164 18247 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:46.634799 18247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908: Generated new TSK 0
I20260927 18:49:46.635742 18247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:49:46.652345 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:46875
--builtin_ntp_servers=127.31.26.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:49:47.008963 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:47.009295 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:47.009479 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:47.017107 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:47.017225 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:47.017405 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:47.026795 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:46875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:47.028168 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:47.030000 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:47.043787 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:47.043867 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:47.043754 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:47.044592 18260 server_base.cc:1034] running on GCE node
I20260927 18:49:47.045389 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:47.046407 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:47.047652 18260 hybrid_clock.cc:648] HybridClock initialized: now 1790534987047603 us; error 46 us; skew 500 ppm
I20260927 18:49:47.048002 18260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:47.050355 18260 webserver.cc:460] Webserver started at http://127.31.26.193:42723/ using document root <none> and password file <none>
I20260927 18:49:47.051086 18260 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:47.051222 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:47.051550 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:47.054631 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "594202a5a25f44e1b4de83c36c5a7434"
format_stamp: "Formatted at 2026-09-27 18:49:47 on dist-test-slave-rfd5"
server_key: "66e7f66c3ec9cd844402c6585f4c7381"
server_key_iv: "f9f439d6188dd16945f3b592f52f1c7b"
server_key_version: "encryptionkey@0"
I20260927 18:49:47.055483 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "594202a5a25f44e1b4de83c36c5a7434"
format_stamp: "Formatted at 2026-09-27 18:49:47 on dist-test-slave-rfd5"
server_key: "66e7f66c3ec9cd844402c6585f4c7381"
server_key_iv: "f9f439d6188dd16945f3b592f52f1c7b"
server_key_version: "encryptionkey@0"
I20260927 18:49:47.061048 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:49:47.064913 18274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:47.066527 18260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:47.066766 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "594202a5a25f44e1b4de83c36c5a7434"
format_stamp: "Formatted at 2026-09-27 18:49:47 on dist-test-slave-rfd5"
server_key: "66e7f66c3ec9cd844402c6585f4c7381"
server_key_iv: "f9f439d6188dd16945f3b592f52f1c7b"
server_key_version: "encryptionkey@0"
I20260927 18:49:47.067022 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:47.092136 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:47.093148 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:47.093467 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:47.094539 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:47.096093 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:47.096227 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:47.096359 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:47.096447 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:47.131033 18260 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:33223
I20260927 18:49:47.131129 18386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:33223 every 8 connection(s)
I20260927 18:49:47.132421 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:47.142707 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18260
I20260927 18:49:47.142903 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:47.143417 31851 external_mini_cluster.cc:1428] Setting key 4ccddc4614e3e7ae6e28ec72756659ab
I20260927 18:49:47.146330 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:47.135451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:35289 (local address 127.31.26.254:46875)
0927 18:49:47.135884 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:47.135889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:47.137255 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 18:49:47.139118 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:47.139124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:47.139226 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:47.139350 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:47.140523 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.141309 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.143353 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.143829 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.143964 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:47.145479 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:47.145487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:47.145493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:47.145515 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:47.145565 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:47.145584 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:47.145672 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:47.145985 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 18:49:47.146072 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":228,"threads_started":1}
I20260927 18:49:47.146870 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:46875
--builtin_ntp_servers=127.31.26.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:49:47.147974 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:47.136121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:46875 (local address 127.31.26.193:35289)
0927 18:49:47.137026 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:47.137062 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:47.138816 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:47.139602 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:47.139611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:47.139710 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:47.140347 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:47.140365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.141487 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.141492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:47.143170 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:47.143186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.144063 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.144069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:47.144295 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:47.145071 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:47.145075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:47.145184 (+ 109us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:47.145291 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:47.145723 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:47.145736 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:47.145745 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:49:47.145993 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 18:49:47.146442 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":273,"threads_started":1}
I20260927 18:49:47.149699 18387 heartbeater.cc:347] Connected to a master server at 127.31.26.254:46875
I20260927 18:49:47.150061 18387 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:47.151000 18387 heartbeater.cc:511] Master 127.31.26.254:46875 requested a full tablet report, sending...
I20260927 18:49:47.153378 18201 ts_manager.cc:194] Registered new tserver with Master: 594202a5a25f44e1b4de83c36c5a7434 (127.31.26.193:33223)
I20260927 18:49:47.155287 18201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.193:35289
W20260927 18:49:47.490480 18391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:47.490806 18391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:47.490895 18391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:47.498360 18391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:47.498487 18391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:47.498651 18391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:47.508244 18391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:46875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:47.509747 18391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:47.511706 18391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:47.525661 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:47.525689 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:47.525730 18399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:47.526295 18391 server_base.cc:1034] running on GCE node
I20260927 18:49:47.527174 18391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:47.528188 18391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:47.529384 18391 hybrid_clock.cc:648] HybridClock initialized: now 1790534987529332 us; error 43 us; skew 500 ppm
I20260927 18:49:47.529796 18391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:47.532454 18391 webserver.cc:460] Webserver started at http://127.31.26.194:41717/ using document root <none> and password file <none>
I20260927 18:49:47.533275 18391 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:47.533372 18391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:47.533727 18391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:47.536801 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "8128b8746b9d400ea14bbc041cc1a292"
format_stamp: "Formatted at 2026-09-27 18:49:47 on dist-test-slave-rfd5"
server_key: "f3db5d444e69cf1c93a4dc4ba874cb10"
server_key_iv: "28b3df70033bfde81d5e2c6a5261e0bf"
server_key_version: "encryptionkey@0"
I20260927 18:49:47.537766 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "8128b8746b9d400ea14bbc041cc1a292"
format_stamp: "Formatted at 2026-09-27 18:49:47 on dist-test-slave-rfd5"
server_key: "f3db5d444e69cf1c93a4dc4ba874cb10"
server_key_iv: "28b3df70033bfde81d5e2c6a5261e0bf"
server_key_version: "encryptionkey@0"
I20260927 18:49:47.543574 18391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:49:47.547451 18405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:47.549062 18391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:47.549397 18391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "8128b8746b9d400ea14bbc041cc1a292"
format_stamp: "Formatted at 2026-09-27 18:49:47 on dist-test-slave-rfd5"
server_key: "f3db5d444e69cf1c93a4dc4ba874cb10"
server_key_iv: "28b3df70033bfde81d5e2c6a5261e0bf"
server_key_version: "encryptionkey@0"
I20260927 18:49:47.549710 18391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:47.589578 18391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:47.590499 18391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:47.590828 18391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:47.591887 18391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:47.593560 18391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:47.593688 18391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:47.593827 18391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:47.593917 18391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:47.628558 18391 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:37133
I20260927 18:49:47.628619 18517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:37133 every 8 connection(s)
I20260927 18:49:47.630085 18391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:47.634260 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18391
I20260927 18:49:47.634452 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:47.634997 31851 external_mini_cluster.cc:1428] Setting key d9f1776e6443e536b98ef661825ee13a
I20260927 18:49:47.638358 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:46875
--builtin_ntp_servers=127.31.26.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:49:47.645973 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:47.632850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:34955 (local address 127.31.26.254:46875)
0927 18:49:47.633030 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:47.633035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:47.634858 (+ 1823us) server_negotiation.cc:407] Connection header received
0927 18:49:47.637613 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:47.637619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:47.637710 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:47.637833 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:47.639383 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.640607 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.642415 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.643187 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.643348 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:47.644854 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:47.644863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:47.644882 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:47.644913 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:47.644961 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:47.644980 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:47.645106 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:47.645425 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 18:49:47.645563 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:49:47.647027 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:47.633432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:46875 (local address 127.31.26.194:34955)
0927 18:49:47.634663 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:47.634705 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:47.637273 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:47.638247 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:47.638261 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:47.638404 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:47.639169 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:47.639189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.640778 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.640782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:47.642241 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:47.642255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:47.643425 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:47.643430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:47.643661 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:47.644474 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:47.644477 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:47.644525 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:47.644626 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:47.645182 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:47.645191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:47.645199 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:49:47.645378 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 18:49:47.645745 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":338,"threads_started":1}
I20260927 18:49:47.648765 18518 heartbeater.cc:347] Connected to a master server at 127.31.26.254:46875
I20260927 18:49:47.649130 18518 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:47.649968 18518 heartbeater.cc:511] Master 127.31.26.254:46875 requested a full tablet report, sending...
I20260927 18:49:47.651652 18201 ts_manager.cc:194] Registered new tserver with Master: 8128b8746b9d400ea14bbc041cc1a292 (127.31.26.194:37133)
I20260927 18:49:47.652878 18201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.194:34955
W20260927 18:49:47.979972 18521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:47.980268 18521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:47.980356 18521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:47.988345 18521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:47.988469 18521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:47.988579 18521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:47.998512 18521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:46875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:47.999807 18521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:48.001674 18521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:48.015982 18527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:48.016124 18526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:48.016098 18529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:48.017066 18521 server_base.cc:1034] running on GCE node
I20260927 18:49:48.017848 18521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:48.018935 18521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:48.020217 18521 hybrid_clock.cc:648] HybridClock initialized: now 1790534988020165 us; error 47 us; skew 500 ppm
I20260927 18:49:48.020576 18521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:48.023110 18521 webserver.cc:460] Webserver started at http://127.31.26.195:42705/ using document root <none> and password file <none>
I20260927 18:49:48.023904 18521 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:48.024060 18521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:48.024412 18521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:48.027619 18521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "efbd517bcb4f436586967f887e0a908c"
format_stamp: "Formatted at 2026-09-27 18:49:48 on dist-test-slave-rfd5"
server_key: "25449b70f7165c308e91be2ef1b78ab0"
server_key_iv: "eca9cd7622c2f7efa04a80b953a066dc"
server_key_version: "encryptionkey@0"
I20260927 18:49:48.028525 18521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "efbd517bcb4f436586967f887e0a908c"
format_stamp: "Formatted at 2026-09-27 18:49:48 on dist-test-slave-rfd5"
server_key: "25449b70f7165c308e91be2ef1b78ab0"
server_key_iv: "eca9cd7622c2f7efa04a80b953a066dc"
server_key_version: "encryptionkey@0"
I20260927 18:49:48.034490 18521 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260927 18:49:48.038762 18535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:48.040410 18521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:48.040614 18521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "efbd517bcb4f436586967f887e0a908c"
format_stamp: "Formatted at 2026-09-27 18:49:48 on dist-test-slave-rfd5"
server_key: "25449b70f7165c308e91be2ef1b78ab0"
server_key_iv: "eca9cd7622c2f7efa04a80b953a066dc"
server_key_version: "encryptionkey@0"
I20260927 18:49:48.040898 18521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:48.070537 18521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:48.071557 18521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:48.071884 18521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:48.072962 18521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:48.074503 18521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:48.074647 18521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:48.074780 18521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:48.074869 18521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:48.109539 18521 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:42539
I20260927 18:49:48.109627 18647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:42539 every 8 connection(s)
I20260927 18:49:48.110950 18521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:48.118405 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18521
I20260927 18:49:48.118623 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:48.119190 31851 external_mini_cluster.cc:1428] Setting key 0f6eb15add3c761aa4bb9404db9da09a
I20260927 18:49:48.127990 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.114367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:34963 (local address 127.31.26.254:46875)
0927 18:49:48.114654 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.114663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.117305 (+ 2642us) server_negotiation.cc:407] Connection header received
0927 18:49:48.119969 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.119975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.120431 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.120560 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:48.121917 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.122683 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.124885 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.125384 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.125526 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.127094 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:48.127106 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:48.127110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:48.127131 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:48.127165 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:48.127177 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:48.127274 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.127514 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.127630 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:49:48.129164 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.115070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:46875 (local address 127.31.26.195:34963)
0927 18:49:48.117058 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.117153 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:48.119607 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:48.120831 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:48.120841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:48.120976 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:48.121738 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.121762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.122862 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.122867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.124724 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.124738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.125623 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.125630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.125947 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.126728 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:48.126730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:48.126779 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:48.126864 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:48.127273 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:48.127287 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:48.127297 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:49:48.127506 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 18:49:48.127945 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":1081,"threads_started":1}
I20260927 18:49:48.130782 18648 heartbeater.cc:347] Connected to a master server at 127.31.26.254:46875
I20260927 18:49:48.131130 18648 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:48.131911 18648 heartbeater.cc:511] Master 127.31.26.254:46875 requested a full tablet report, sending...
I20260927 18:49:48.133541 18201 ts_manager.cc:194] Registered new tserver with Master: efbd517bcb4f436586967f887e0a908c (127.31.26.195:42539)
I20260927 18:49:48.134685 18201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.195:34963
I20260927 18:49:48.135426 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:48.145483 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.139549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.31.26.254:46875)
0927 18:49:48.139817 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.139824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.140070 (+ 246us) server_negotiation.cc:407] Connection header received
0927 18:49:48.140221 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.140225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.140288 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.140371 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:48.140827 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.141641 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.142981 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.143391 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.143494 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.144678 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:48.144683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:48.144686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:48.144705 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:48.144738 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:48.144750 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:48.144882 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.145033 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.145127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:49:48.149866 18201 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:48.156924 18201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:48.158918 18387 heartbeater.cc:503] Master 127.31.26.254:46875 was elected leader, sending a full tablet report...
I20260927 18:49:48.192519 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.173113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.31.26.195:42539)
0927 18:49:48.181745 (+ 8632us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.181758 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.184483 (+ 2725us) server_negotiation.cc:407] Connection header received
0927 18:49:48.184718 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.184725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.184924 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.185153 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:48.186124 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.187587 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.190075 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.191305 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.191472 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.191799 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.191989 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.192166 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8390,"thread_start_us":7952,"threads_started":1}
I20260927 18:49:48.192576 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.172459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:42539 (local address 127.0.0.1:34840)
0927 18:49:48.184341 (+ 11882us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.184357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:48.184528 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:48.185190 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:48.185197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:48.185224 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:48.185899 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.185912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.187730 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.187738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.189825 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.189841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.191510 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.191515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.191717 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.191773 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 18:49:48.191933 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 18:49:48.192142 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11762,"mutex_wait_us":115,"thread_start_us":11574,"threads_started":1}
I20260927 18:49:48.193570 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.170351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37600)
0927 18:49:48.171391 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.171474 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:48.171758 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:48.184558 (+ 12800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:48.184564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:48.184595 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:48.185175 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.185190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.187335 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.187341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.190170 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.190184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.192584 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.192590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.192822 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.192953 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 18:49:48.193039 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:49:48.193222 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":297,"thread_start_us":996,"threads_started":1}
I20260927 18:49:48.193688 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.170675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.31.26.193:33223)
0927 18:49:48.183901 (+ 13226us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.183910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.183940 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:49:48.184040 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.184047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.184331 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.184894 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:48.185469 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.187219 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.190406 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.191969 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.192123 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.192425 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.193169 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.193351 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13040,"thread_start_us":12888,"threads_started":1}
I20260927 18:49:48.193738 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.171122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.31.26.194:37133)
0927 18:49:48.182079 (+ 10957us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.182092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.183179 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 18:49:48.183314 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.183323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.183647 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.183877 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:48.185120 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.187141 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.190713 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.192079 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.192195 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.192396 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.193180 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.193355 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10716,"thread_start_us":10998,"threads_started":1}
I20260927 18:49:48.194401 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.170898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:37133 (local address 127.0.0.1:36192)
0927 18:49:48.182841 (+ 11943us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.182863 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:48.183016 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:48.184196 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:48.184204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:48.184248 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:48.184869 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.184882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.187288 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.187296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.190487 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.190499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.192570 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.192578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.192821 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.192955 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 18:49:48.193039 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:49:48.194020 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11807,"mutex_wait_us":1180,"thread_start_us":11639,"threads_started":1}
I20260927 18:49:48.196978 18583 tablet_service.cc:1467] Processing CreateTablet for tablet f0f9092a99454d95b6c0fa635f85ecdc (DEFAULT_TABLE table=test-table [id=a80d67772ecf49088767b00edd281f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:48.198537 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f9092a99454d95b6c0fa635f85ecdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:48.198292 18322 tablet_service.cc:1467] Processing CreateTablet for tablet f0f9092a99454d95b6c0fa635f85ecdc (DEFAULT_TABLE table=test-table [id=a80d67772ecf49088767b00edd281f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:48.198294 18453 tablet_service.cc:1467] Processing CreateTablet for tablet f0f9092a99454d95b6c0fa635f85ecdc (DEFAULT_TABLE table=test-table [id=a80d67772ecf49088767b00edd281f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:48.199708 18322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f9092a99454d95b6c0fa635f85ecdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:48.199707 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f9092a99454d95b6c0fa635f85ecdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:48.210592 18666 tablet_bootstrap.cc:492] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Bootstrap starting.
I20260927 18:49:48.214725 18668 tablet_bootstrap.cc:492] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: Bootstrap starting.
I20260927 18:49:48.215147 18666 tablet_bootstrap.cc:654] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:48.215580 18667 tablet_bootstrap.cc:492] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Bootstrap starting.
I20260927 18:49:48.217002 18666 log.cc:824] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:48.219167 18668 tablet_bootstrap.cc:654] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:48.219213 18667 tablet_bootstrap.cc:654] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:48.220331 18668 log.cc:824] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:48.220464 18667 log.cc:824] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:48.220731 18666 tablet_bootstrap.cc:492] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: No bootstrap required, opened a new log
I20260927 18:49:48.221058 18666 ts_tablet_manager.cc:1397] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:49:48.223794 18668 tablet_bootstrap.cc:492] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: No bootstrap required, opened a new log
I20260927 18:49:48.224123 18668 ts_tablet_manager.cc:1397] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260927 18:49:48.230118 18666 raft_consensus.cc:348] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.230635 18666 raft_consensus.cc:374] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:48.230799 18666 raft_consensus.cc:729] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbd517bcb4f436586967f887e0a908c, State: Initialized, Role: FOLLOWER
I20260927 18:49:48.231736 18667 tablet_bootstrap.cc:492] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: No bootstrap required, opened a new log
I20260927 18:49:48.231535 18666 consensus_queue.cc:260] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.232062 18667 ts_tablet_manager.cc:1397] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260927 18:49:48.232159 18668 raft_consensus.cc:348] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.232698 18668 raft_consensus.cc:374] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:48.232764 18648 heartbeater.cc:503] Master 127.31.26.254:46875 was elected leader, sending a full tablet report...
I20260927 18:49:48.232822 18668 raft_consensus.cc:729] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594202a5a25f44e1b4de83c36c5a7434, State: Initialized, Role: FOLLOWER
I20260927 18:49:48.233646 18668 consensus_queue.cc:260] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.237859 18666 ts_tablet_manager.cc:1428] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Time spent starting tablet: real 0.017s user 0.004s sys 0.008s
I20260927 18:49:48.239652 18668 ts_tablet_manager.cc:1428] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260927 18:49:48.242769 18667 raft_consensus.cc:348] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.243185 18667 raft_consensus.cc:374] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:48.243317 18667 raft_consensus.cc:729] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8128b8746b9d400ea14bbc041cc1a292, State: Initialized, Role: FOLLOWER
I20260927 18:49:48.243896 18667 consensus_queue.cc:260] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.245008 18518 heartbeater.cc:503] Master 127.31.26.254:46875 was elected leader, sending a full tablet report...
I20260927 18:49:48.245687 18667 ts_tablet_manager.cc:1428] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260927 18:49:48.280653 18674 raft_consensus.cc:482] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:48.281008 18674 raft_consensus.cc:504] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.282763 18674 leader_election.cc:288] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 594202a5a25f44e1b4de83c36c5a7434 (127.31.26.193:33223), efbd517bcb4f436586967f887e0a908c (127.31.26.195:42539)
I20260927 18:49:48.291860 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.284439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:41309 (local address 127.31.26.195:42539)
0927 18:49:48.284676 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.284685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.285189 (+ 504us) server_negotiation.cc:407] Connection header received
0927 18:49:48.285349 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.285352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.285416 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.285535 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:48.285998 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.286978 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.289372 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.290587 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.290746 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.290823 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.291457 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.291595 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:49:48.291860 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.283258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:42477 (local address 127.31.26.193:33223)
0927 18:49:48.283483 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.283488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.284139 (+ 651us) server_negotiation.cc:407] Connection header received
0927 18:49:48.284448 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.284453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.284549 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.284631 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:48.285511 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.286491 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.289257 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.290533 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.290746 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.290831 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.291464 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.291595 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:49:48.292030 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.283225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.31.26.194:42477)
0927 18:49:48.284012 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.284032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:48.284273 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:48.284862 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:48.284867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:48.284894 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:48.285376 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.285386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.286632 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.286635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.288971 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.288987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.290874 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.290879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.291215 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.291301 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:49:48.291409 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:49:48.291607 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":171,"thread_start_us":393,"threads_started":1}
I20260927 18:49:48.292047 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.284117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:42539 (local address 127.31.26.194:41309)
0927 18:49:48.285052 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.285069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:48.285222 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:48.285564 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:48.285567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:48.285586 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:48.285856 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.285863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.287109 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.287113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.289177 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:48.289192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.290868 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.290876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:48.291208 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.291301 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 18:49:48.291423 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 18:49:48.291607 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":84,"thread_start_us":573,"threads_started":1}
I20260927 18:49:48.293417 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9092a99454d95b6c0fa635f85ecdc" candidate_uuid: "8128b8746b9d400ea14bbc041cc1a292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbd517bcb4f436586967f887e0a908c" is_pre_election: true
I20260927 18:49:48.293433 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9092a99454d95b6c0fa635f85ecdc" candidate_uuid: "8128b8746b9d400ea14bbc041cc1a292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594202a5a25f44e1b4de83c36c5a7434" is_pre_election: true
I20260927 18:49:48.294061 18603 raft_consensus.cc:2454] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8128b8746b9d400ea14bbc041cc1a292 in term 0.
I20260927 18:49:48.294061 18342 raft_consensus.cc:2454] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8128b8746b9d400ea14bbc041cc1a292 in term 0.
I20260927 18:49:48.294821 18408 leader_election.cc:302] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8128b8746b9d400ea14bbc041cc1a292, efbd517bcb4f436586967f887e0a908c; no voters:
I20260927 18:49:48.295214 18674 raft_consensus.cc:2781] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:48.295373 18674 raft_consensus.cc:482] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:48.295512 18674 raft_consensus.cc:3037] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:48.297816 18674 raft_consensus.cc:504] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.298446 18674 leader_election.cc:288] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [CANDIDATE]: Term 1 election: Requested vote from peers 594202a5a25f44e1b4de83c36c5a7434 (127.31.26.193:33223), efbd517bcb4f436586967f887e0a908c (127.31.26.195:42539)
I20260927 18:49:48.299068 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9092a99454d95b6c0fa635f85ecdc" candidate_uuid: "8128b8746b9d400ea14bbc041cc1a292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594202a5a25f44e1b4de83c36c5a7434"
I20260927 18:49:48.299067 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f9092a99454d95b6c0fa635f85ecdc" candidate_uuid: "8128b8746b9d400ea14bbc041cc1a292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbd517bcb4f436586967f887e0a908c"
I20260927 18:49:48.299304 18603 raft_consensus.cc:3037] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:48.299304 18342 raft_consensus.cc:3037] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:48.302657 18342 raft_consensus.cc:2454] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8128b8746b9d400ea14bbc041cc1a292 in term 1.
I20260927 18:49:48.302657 18603 raft_consensus.cc:2454] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8128b8746b9d400ea14bbc041cc1a292 in term 1.
I20260927 18:49:48.303157 18408 leader_election.cc:302] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 594202a5a25f44e1b4de83c36c5a7434, 8128b8746b9d400ea14bbc041cc1a292; no voters:
I20260927 18:49:48.303476 18674 raft_consensus.cc:2781] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:48.303941 18674 raft_consensus.cc:686] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 1 LEADER]: Becoming Leader. State: Replica: 8128b8746b9d400ea14bbc041cc1a292, State: Running, Role: LEADER
I20260927 18:49:48.304442 18674 consensus_queue.cc:237] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } }
I20260927 18:49:48.309595 18200 catalog_manager.cc:5495] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8128b8746b9d400ea14bbc041cc1a292 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "8128b8746b9d400ea14bbc041cc1a292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8128b8746b9d400ea14bbc041cc1a292" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 37133 } health_report { overall_health: HEALTHY } } }
I20260927 18:49:48.351882 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.346544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.31.26.194:37133)
0927 18:49:48.346757 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.346763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.346792 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:49:48.346878 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.346882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.346977 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.347112 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:48.347508 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.348202 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.349639 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.350050 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.350172 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.350664 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:48.351190 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:48.351319 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.351490 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.351562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:49:48.361020 18342 raft_consensus.cc:1261] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 1 FOLLOWER]: Refusing update from remote peer 8128b8746b9d400ea14bbc041cc1a292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:48.361019 18603 raft_consensus.cc:1261] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 1 FOLLOWER]: Refusing update from remote peer 8128b8746b9d400ea14bbc041cc1a292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:48.362332 18677 consensus_queue.cc:1035] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbd517bcb4f436586967f887e0a908c" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:48.362848 18674 consensus_queue.cc:1035] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "594202a5a25f44e1b4de83c36c5a7434" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:49:48.363212 18649 tablet.cc:2369] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:48.392776 18201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:48.393195 18201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:48.397373 18201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e0ca64961d945b4aea4771e8eeed908: Sending DeleteTablet for 3 replicas of tablet f0f9092a99454d95b6c0fa635f85ecdc
I20260927 18:49:48.398756 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f9092a99454d95b6c0fa635f85ecdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:48 UTC) from {username='slave'} at 127.0.0.1:34840
I20260927 18:49:48.398761 18322 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f9092a99454d95b6c0fa635f85ecdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:48 UTC) from {username='slave'} at 127.0.0.1:37600
I20260927 18:49:48.398931 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet f0f9092a99454d95b6c0fa635f85ecdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:48 UTC) from {username='slave'} at 127.0.0.1:36192
I20260927 18:49:48.400379 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18260
I20260927 18:49:48.413997 18690 tablet_replica.cc:330] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: stopping tablet replica
I20260927 18:49:48.414680 18690 raft_consensus.cc:2229] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:48.415422 18690 raft_consensus.cc:2258] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:48.416131 18692 tablet_replica.cc:330] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: stopping tablet replica
I20260927 18:49:48.419466 18690 ts_tablet_manager.cc:1905] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:48.417073 18692 raft_consensus.cc:2229] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:48.421020 18692 raft_consensus.cc:2258] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:48.423730 18692 ts_tablet_manager.cc:1905] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:48.424836 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.402426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.31.26.193:33223)
0927 18:49:48.402697 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.402707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.412831 (+ 10124us) server_negotiation.cc:407] Connection header received
0927 18:49:48.413076 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.413113 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.413316 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.413468 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:48.414353 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.416017 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.418842 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.419833 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.419973 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.422303 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:48.422321 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:48.422346 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:48.423477 (+ 1131us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:48.423573 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:48.423607 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:48.423723 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.424086 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.424277 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:49:48.426416 18691 tablet_replica.cc:330] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434: stopping tablet replica
I20260927 18:49:48.426545 18382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:48.431985 18690 ts_tablet_manager.cc:1918] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:48.432283 18690 log.cc:1196] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/f0f9092a99454d95b6c0fa635f85ecdc
I20260927 18:49:48.432687 18692 ts_tablet_manager.cc:1918] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:48.432991 18690 ts_tablet_manager.cc:1939] T f0f9092a99454d95b6c0fa635f85ecdc P efbd517bcb4f436586967f887e0a908c: Deleting consensus metadata
I20260927 18:49:48.433115 18692 log.cc:1196] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/f0f9092a99454d95b6c0fa635f85ecdc
I20260927 18:49:48.433863 18692 ts_tablet_manager.cc:1939] T f0f9092a99454d95b6c0fa635f85ecdc P 8128b8746b9d400ea14bbc041cc1a292: Deleting consensus metadata
I20260927 18:49:48.435418 18187 catalog_manager.cc:4841] TS efbd517bcb4f436586967f887e0a908c (127.31.26.195:42539): tablet f0f9092a99454d95b6c0fa635f85ecdc (table test-table [id=a80d67772ecf49088767b00edd281f66]) successfully deleted
I20260927 18:49:48.436221 18185 catalog_manager.cc:4841] TS 8128b8746b9d400ea14bbc041cc1a292 (127.31.26.194:37133): tablet f0f9092a99454d95b6c0fa635f85ecdc (table test-table [id=a80d67772ecf49088767b00edd281f66]) successfully deleted
I20260927 18:49:48.427131 18691 raft_consensus.cc:2229] T f0f9092a99454d95b6c0fa635f85ecdc P 594202a5a25f44e1b4de83c36c5a7434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:48.542487 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18260
W20260927 18:49:48.561486 18187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.26.193:33223 (error 108)
I20260927 18:49:48.561751 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18391
I20260927 18:49:48.563802 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.563327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37624)
0927 18:49:48.563504 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.563622 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:49:48.564114 18187 catalog_manager.cc:4584] TS 594202a5a25f44e1b4de83c36c5a7434 (127.31.26.193:33223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0f9092a99454d95b6c0fa635f85ecdc: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
I20260927 18:49:48.570689 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.562680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.31.26.194:37133)
0927 18:49:48.562883 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.562890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.563023 (+ 133us) server_negotiation.cc:407] Connection header received
0927 18:49:48.563215 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.563227 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.563340 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.563480 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:48.564061 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.564981 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.567041 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.567496 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.567595 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.568883 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:48.568893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:48.568907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:48.569705 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:48.569759 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:48.569781 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:48.569874 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.570112 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.570206 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:49:48.571429 18513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:48.617813 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.617269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37640)
0927 18:49:48.617502 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.617616 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 18:49:48.618924 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.618510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37652)
0927 18:49:48.618681 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.618769 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 18:49:48.679912 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.679332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37656)
0927 18:49:48.679585 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.679696 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 18:49:48.681838 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.680881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37664)
0927 18:49:48.681608 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.681698 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475}
I20260927 18:49:48.687879 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18391
I20260927 18:49:48.706458 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18521
I20260927 18:49:48.712940 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.707219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.31.26.195:42539)
0927 18:49:48.707389 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:48.707393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:48.707413 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:49:48.707594 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:48.707598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:48.707676 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:48.707763 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:48.708112 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.708844 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.710145 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:48.710530 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:48.710622 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:48.711758 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:48.711768 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:48.711783 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:48.712352 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:48.712409 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:48.712428 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:48.712520 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:48.712607 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 18:49:48.712698 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:49:48.713654 18643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:48.762800 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.762307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37670)
0927 18:49:48.762562 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.762652 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 18:49:48.763875 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.763482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37684)
0927 18:49:48.763684 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.763752 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 18:49:48.803350 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18521
I20260927 18:49:48.820695 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18170
I20260927 18:49:48.822008 18231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:48.936738 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:48.935940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33223 (local address 127.0.0.1:37694)
0927 18:49:48.936281 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:48.936473 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 18:49:48.936905 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18170
2026-09-27T18:49:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:49 dist-test-slave-rfd5 krb5kdc[18701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:49 dist-test-slave-rfd5 krb5kdc[18701](info): set up 2 sockets
Sep 27 18:49:49 dist-test-slave-rfd5 krb5kdc[18701](info): commencing operation
krb5kdc: starting...
W20260927 18:49:50.065546 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:49:50 dist-test-slave-rfd5 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:49:50.332873 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33165
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:50.685808 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:50.686194 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:50.686362 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:50.693993 18714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:50.694172 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:50.694290 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:50.694397 18714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:50.694494 18714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:50.704715 18714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:50.706216 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:50.707648 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:50.720115 18720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:50.720067 18722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:50.720103 18719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:50.720940 18714 server_base.cc:1034] running on GCE node
I20260927 18:49:50.721825 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:50.723428 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:50.724684 18714 hybrid_clock.cc:648] HybridClock initialized: now 1790534990724619 us; error 55 us; skew 500 ppm
I20260927 18:49:50.725209 18714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:50 dist-test-slave-rfd5 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:50.728706 18714 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:49:50.730602 18714 webserver.cc:460] Webserver started at http://127.31.26.254:38435/ using document root <none> and password file <none>
I20260927 18:49:50.731407 18714 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:50.731555 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:50.731904 18714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:50.735427 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "67952e48b1d249a298fc4e53a68cffe9"
format_stamp: "Formatted at 2026-09-27 18:49:50 on dist-test-slave-rfd5"
server_key: "2f75e3516aef3009568d02acc13fa61a"
server_key_iv: "2a30e9bb83d8221f5f819d9373931500"
server_key_version: "encryptionkey@0"
I20260927 18:49:50.736393 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "67952e48b1d249a298fc4e53a68cffe9"
format_stamp: "Formatted at 2026-09-27 18:49:50 on dist-test-slave-rfd5"
server_key: "2f75e3516aef3009568d02acc13fa61a"
server_key_iv: "2a30e9bb83d8221f5f819d9373931500"
server_key_version: "encryptionkey@0"
I20260927 18:49:50.742745 18714 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:49:50.746935 18729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:50.748605 18714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:49:50.748826 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "67952e48b1d249a298fc4e53a68cffe9"
format_stamp: "Formatted at 2026-09-27 18:49:50 on dist-test-slave-rfd5"
server_key: "2f75e3516aef3009568d02acc13fa61a"
server_key_iv: "2a30e9bb83d8221f5f819d9373931500"
server_key_version: "encryptionkey@0"
I20260927 18:49:50.749073 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:50.762320 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:50.767158 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:50.767517 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:50.787315 18714 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:33165
I20260927 18:49:50.787335 18780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:33165 every 8 connection(s)
I20260927 18:49:50.788823 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:50.789291 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18714
I20260927 18:49:50.789474 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:50.790051 31851 external_mini_cluster.cc:1428] Setting key 055fc97b40c51a237ca72886eb158c30
I20260927 18:49:50.793593 18781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:49:50 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:50.804647 18781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9: Bootstrap starting.
I20260927 18:49:50.808123 18781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:50.809350 18781 log.cc:824] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:50.812892 18781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9: No bootstrap required, opened a new log
I20260927 18:49:50.813422 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:50.792082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.31.26.254:33165)
0927 18:49:50.792911 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:50.792924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:50.792991 (+ 67us) server_negotiation.cc:407] Connection header received
0927 18:49:50.794873 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:50.794896 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:50.795295 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:50.795686 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:50.797235 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:50.798603 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:50.799943 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:50.800640 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:50.800845 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:50.805197 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:50.805207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:50.805223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:50.805266 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:50.808620 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:50.809546 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:50.809557 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:50.809568 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:50.809802 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:50.810398 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:50.810404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:50.810410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:50.811109 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:50.811338 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:50.811832 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 18:49:50.812278 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":360,"threads_started":1}
I20260927 18:49:50.821738 18781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67952e48b1d249a298fc4e53a68cffe9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33165 } }
I20260927 18:49:50.822243 18781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:50.822359 18781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67952e48b1d249a298fc4e53a68cffe9, State: Initialized, Role: FOLLOWER
I20260927 18:49:50.822957 18781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67952e48b1d249a298fc4e53a68cffe9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33165 } }
I20260927 18:49:50.823231 18781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:50.823385 18781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:50.823567 18781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:50.826079 18781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67952e48b1d249a298fc4e53a68cffe9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33165 } }
I20260927 18:49:50.826740 18781 leader_election.cc:302] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67952e48b1d249a298fc4e53a68cffe9; no voters:
I20260927 18:49:50.827342 18781 leader_election.cc:288] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:50.827485 18786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:50.828218 18786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [term 1 LEADER]: Becoming Leader. State: Replica: 67952e48b1d249a298fc4e53a68cffe9, State: Running, Role: LEADER
I20260927 18:49:50.828763 18786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67952e48b1d249a298fc4e53a68cffe9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33165 } }
I20260927 18:49:50.829620 18781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:50.831826 18788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67952e48b1d249a298fc4e53a68cffe9. Latest consensus state: current_term: 1 leader_uuid: "67952e48b1d249a298fc4e53a68cffe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67952e48b1d249a298fc4e53a68cffe9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33165 } } }
I20260927 18:49:50.832332 18788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:50.831816 18787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67952e48b1d249a298fc4e53a68cffe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67952e48b1d249a298fc4e53a68cffe9" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 33165 } } }
I20260927 18:49:50.832504 18787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:50.835273 18793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:50.841188 18793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:50.850781 18793 catalog_manager.cc:1347] Generated new cluster ID: dad3f218576f4f42826b9e6e35e396e6
I20260927 18:49:50.850942 18793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:50.882807 18793 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:50.884306 18793 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:50.892422 18793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9: Generated new TSK 0
I20260927 18:49:50.893499 18793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.26.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
Entry for principal kudu/127.31.26.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab.
I20260927 18:49:50.961565 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33165
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:51.310254 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:51.310559 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:51.310717 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:51.319595 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:51.319718 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:51.319934 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:51.330262 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.193.keytab
--principal=kudu/127.31.26.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:51.331645 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:51.333025 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:51.347391 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:51.347282 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:51.347867 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:51.348589 18807 server_base.cc:1034] running on GCE node
I20260927 18:49:51.349439 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:51.350463 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:51.351717 18807 hybrid_clock.cc:648] HybridClock initialized: now 1790534991351669 us; error 40 us; skew 500 ppm
I20260927 18:49:51.352099 18807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:51 dist-test-slave-rfd5 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534991, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:51.355453 18807 init.cc:377] Logged in from keytab as kudu/127.31.26.193@KRBTEST.COM (short username kudu)
I20260927 18:49:51.356982 18807 webserver.cc:460] Webserver started at http://127.31.26.193:37319/ using document root <none> and password file <none>
I20260927 18:49:51.357724 18807 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:51.357831 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:51.358099 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:51.361133 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "ba9da5bbef27460d84838fa77299d1f8"
format_stamp: "Formatted at 2026-09-27 18:49:51 on dist-test-slave-rfd5"
server_key: "2bb579f273ac8ef15dbbf0bb870ccec3"
server_key_iv: "291e20dab6f6aa911b3552d23ed3b30b"
server_key_version: "encryptionkey@0"
I20260927 18:49:51.361914 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9da5bbef27460d84838fa77299d1f8"
format_stamp: "Formatted at 2026-09-27 18:49:51 on dist-test-slave-rfd5"
server_key: "2bb579f273ac8ef15dbbf0bb870ccec3"
server_key_iv: "291e20dab6f6aa911b3552d23ed3b30b"
server_key_version: "encryptionkey@0"
I20260927 18:49:51.367579 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:49:51.371438 18822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:51.372937 18807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:51.373175 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "ba9da5bbef27460d84838fa77299d1f8"
format_stamp: "Formatted at 2026-09-27 18:49:51 on dist-test-slave-rfd5"
server_key: "2bb579f273ac8ef15dbbf0bb870ccec3"
server_key_iv: "291e20dab6f6aa911b3552d23ed3b30b"
server_key_version: "encryptionkey@0"
I20260927 18:49:51.373399 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:51.389729 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:51.394399 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:51.394761 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:51.395771 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:51.397199 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:51.397279 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:51.397464 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:51.397526 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:51.431614 18807 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:44801
I20260927 18:49:51.431669 18934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:44801 every 8 connection(s)
I20260927 18:49:51.433002 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:51.438043 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18807
I20260927 18:49:51.438263 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:51.438830 31851 external_mini_cluster.cc:1428] Setting key 019f53d85986a4db7791da91ad26e4e9
Sep 27 18:49:51 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534991, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.193@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:51.462322 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:51.436025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:35675 (local address 127.31.26.254:33165)
0927 18:49:51.436555 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:51.436562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:51.437615 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 18:49:51.440126 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:51.440130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:51.440193 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:51.440315 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:51.442422 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.443163 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.445710 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.446109 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.446235 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:51.454580 (+ 8345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:51.454590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:51.454598 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:51.454668 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:51.458604 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:51.459493 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:51.459503 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:51.459509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:51.459678 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:51.460132 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:51.460141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:51.460147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:51.460554 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:51.460725 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:51.461538 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 18:49:51.461817 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":243,"threads_started":1}
I20260927 18:49:51.463102 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:51.436690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33165 (local address 127.31.26.193:35675)
0927 18:49:51.437448 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:51.437494 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:51.439808 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:51.440627 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:51.440639 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:51.441374 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:51.442208 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:51.442236 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.443339 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.443347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:51.445530 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:51.445548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.446375 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.446385 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:51.446749 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:51.451455 (+ 4706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:51.451482 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:51.454257 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:51.458813 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:51.458821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:51.458838 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:51.459329 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:51.459796 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:51.459800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:51.459802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:51.459995 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:51.460684 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:51.460690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:51.461045 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 18:49:51.461371 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 18:49:51.461810 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":334,"threads_started":1}
I20260927 18:49:51.465065 18935 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33165
I20260927 18:49:51.465485 18935 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.26.194@KRBTEST.COM; defaulting to no policy
I20260927 18:49:51.466544 18935 heartbeater.cc:511] Master 127.31.26.254:33165 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.194@KRBTEST.COM" created.
I20260927 18:49:51.469250 18746 ts_manager.cc:194] Registered new tserver with Master: ba9da5bbef27460d84838fa77299d1f8 (127.31.26.193:44801)
I20260927 18:49:51.471601 18746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.193@KRBTEST.COM'} at 127.31.26.193:35675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
Entry for principal kudu/127.31.26.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab.
I20260927 18:49:51.501010 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33165
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:51.844686 18941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:51.845060 18941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:51.845269 18941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:51.852898 18941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:51.853125 18941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:51.853322 18941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:51.863603 18941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.194.keytab
--principal=kudu/127.31.26.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:51.865244 18941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:51.866673 18941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:51.880867 18947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:51.880892 18949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:51.880946 18946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:51.881754 18941 server_base.cc:1034] running on GCE node
I20260927 18:49:51.882464 18941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:51.883535 18941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:51.884770 18941 hybrid_clock.cc:648] HybridClock initialized: now 1790534991884736 us; error 39 us; skew 500 ppm
I20260927 18:49:51.885226 18941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:51 dist-test-slave-rfd5 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534991, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:51.888828 18941 init.cc:377] Logged in from keytab as kudu/127.31.26.194@KRBTEST.COM (short username kudu)
I20260927 18:49:51.890533 18941 webserver.cc:460] Webserver started at http://127.31.26.194:36071/ using document root <none> and password file <none>
I20260927 18:49:51.891239 18941 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:51.891345 18941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:51.891613 18941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:51.894805 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "a37108b142fb4046b4b0142a64d90729"
format_stamp: "Formatted at 2026-09-27 18:49:51 on dist-test-slave-rfd5"
server_key: "167b86989ebcc96a83e0f7c3b0ae4152"
server_key_iv: "838e68d2b31764c2d9e1d5a1344b7cb5"
server_key_version: "encryptionkey@0"
I20260927 18:49:51.895759 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "a37108b142fb4046b4b0142a64d90729"
format_stamp: "Formatted at 2026-09-27 18:49:51 on dist-test-slave-rfd5"
server_key: "167b86989ebcc96a83e0f7c3b0ae4152"
server_key_iv: "838e68d2b31764c2d9e1d5a1344b7cb5"
server_key_version: "encryptionkey@0"
I20260927 18:49:51.901505 18941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:49:51.905392 18956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:51.906973 18941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:51.907186 18941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "a37108b142fb4046b4b0142a64d90729"
format_stamp: "Formatted at 2026-09-27 18:49:51 on dist-test-slave-rfd5"
server_key: "167b86989ebcc96a83e0f7c3b0ae4152"
server_key_iv: "838e68d2b31764c2d9e1d5a1344b7cb5"
server_key_version: "encryptionkey@0"
I20260927 18:49:51.907437 18941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:51.926929 18941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:51.931874 18941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:51.932185 18941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:51.933337 18941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:51.934744 18941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:51.934820 18941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:51.934911 18941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:51.934954 18941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:51.969355 18941 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:39267
I20260927 18:49:51.969391 19068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:39267 every 8 connection(s)
I20260927 18:49:51.970755 18941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:51.977037 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 18941
I20260927 18:49:51.977255 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:51.977839 31851 external_mini_cluster.cc:1428] Setting key 3c51acb2b496e340a9cadde99a846b78
Sep 27 18:49:51 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534991, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.194@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:51.992169 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:51.973541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:42899 (local address 127.31.26.254:33165)
0927 18:49:51.974012 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:51.974016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:51.974984 (+ 968us) server_negotiation.cc:407] Connection header received
0927 18:49:51.977128 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:51.977138 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:51.977253 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:51.977398 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:51.979260 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.980112 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.981865 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.982316 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.982458 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:51.986213 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:51.986219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:51.986222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:51.986258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:51.988912 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:51.989727 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:51.989734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:51.989738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:51.989881 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:51.990343 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:51.990348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:51.990353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:51.990634 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:51.990794 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:51.991539 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 18:49:51.991779 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260927 18:49:51.993290 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:51.974036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33165 (local address 127.31.26.194:42899)
0927 18:49:51.974816 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:51.974855 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:51.976707 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:51.977718 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:51.977727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:51.978341 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:51.979069 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:51.979089 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.980289 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.980297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:51.981702 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:51.981715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:51.982602 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:51.982611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:51.982864 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:51.983598 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:51.983633 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:51.985932 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:51.989071 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:51.989102 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:51.989129 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:51.989584 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:51.990007 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:51.990012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:51.990016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:51.990219 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:51.990765 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:51.990774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:51.991148 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 18:49:51.991441 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 18:49:51.992035 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":275,"threads_started":1}
I20260927 18:49:51.994884 19069 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33165
I20260927 18:49:51.995231 19069 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:51.996202 19069 heartbeater.cc:511] Master 127.31.26.254:33165 requested a full tablet report, sending...
I20260927 18:49:51.998059 18746 ts_manager.cc:194] Registered new tserver with Master: a37108b142fb4046b4b0142a64d90729 (127.31.26.194:39267)
I20260927 18:49:51.999377 18746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.194@KRBTEST.COM'} at 127.31.26.194:42899
WARNING: no policy specified for kudu/127.31.26.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
Entry for principal kudu/127.31.26.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab.
I20260927 18:49:52.036860 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33165
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:35431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:52.388156 19075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:52.388543 19075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:52.388722 19075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:52.396261 19075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:52.396461 19075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:52.396662 19075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:52.407724 19075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:35431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.195.keytab
--principal=kudu/127.31.26.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:33165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:52.409204 19075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:52.410604 19075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:52.424713 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:52.424664 19081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:52.424618 19083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:52.425889 19075 server_base.cc:1034] running on GCE node
I20260927 18:49:52.426626 19075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:52.427660 19075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:52.428903 19075 hybrid_clock.cc:648] HybridClock initialized: now 1790534992428854 us; error 47 us; skew 500 ppm
I20260927 18:49:52.429314 19075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:52 dist-test-slave-rfd5 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:52.432662 19075 init.cc:377] Logged in from keytab as kudu/127.31.26.195@KRBTEST.COM (short username kudu)
I20260927 18:49:52.434459 19075 webserver.cc:460] Webserver started at http://127.31.26.195:37805/ using document root <none> and password file <none>
I20260927 18:49:52.435199 19075 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:52.435338 19075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:52.435684 19075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:52.438817 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "82df159762b74a47a9779cc858fb14c1"
format_stamp: "Formatted at 2026-09-27 18:49:52 on dist-test-slave-rfd5"
server_key: "442c45762ba76d6c607c8a047e75f748"
server_key_iv: "09b908ccc4f77422cbdcfe54c20523ec"
server_key_version: "encryptionkey@0"
I20260927 18:49:52.439621 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "82df159762b74a47a9779cc858fb14c1"
format_stamp: "Formatted at 2026-09-27 18:49:52 on dist-test-slave-rfd5"
server_key: "442c45762ba76d6c607c8a047e75f748"
server_key_iv: "09b908ccc4f77422cbdcfe54c20523ec"
server_key_version: "encryptionkey@0"
I20260927 18:49:52.445067 19075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:49:52.448807 19090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:52.450385 19075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:52.450600 19075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "82df159762b74a47a9779cc858fb14c1"
format_stamp: "Formatted at 2026-09-27 18:49:52 on dist-test-slave-rfd5"
server_key: "442c45762ba76d6c607c8a047e75f748"
server_key_iv: "09b908ccc4f77422cbdcfe54c20523ec"
server_key_version: "encryptionkey@0"
I20260927 18:49:52.450821 19075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:52.468601 19075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:52.473564 19075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:52.473876 19075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:52.475024 19075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:52.475312 18935 heartbeater.cc:503] Master 127.31.26.254:33165 was elected leader, sending a full tablet report...
I20260927 18:49:52.476881 19075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:52.477034 19075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:49:52.477213 19075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:52.477296 19075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:52.511859 19075 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:35063
I20260927 18:49:52.511883 19202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:35063 every 8 connection(s)
I20260927 18:49:52.513387 19075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:52.522742 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19075
I20260927 18:49:52.522955 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:52.523422 31851 external_mini_cluster.cc:1428] Setting key 6e066f5c018d47464a56a02e545fdd62
Sep 27 18:49:52 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.195@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:49:52.536392 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.516205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:42401 (local address 127.31.26.254:33165)
0927 18:49:52.516693 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.516698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.517689 (+ 991us) server_negotiation.cc:407] Connection header received
0927 18:49:52.519704 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.519710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.519808 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.519935 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:52.521782 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.522830 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.525023 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.525681 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.525783 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.529881 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:52.529886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:52.529889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:52.529927 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:52.532930 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:52.533859 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:52.533863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:52.533865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:52.534001 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:52.534507 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:52.534511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:52.534513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:52.534855 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:52.535064 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.535835 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.536022 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260927 18:49:52.537572 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.516697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:33165 (local address 127.31.26.195:42401)
0927 18:49:52.517499 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:52.517541 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:52.519346 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:52.520218 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:52.520228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:52.520795 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:52.521586 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.521606 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.523118 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.523124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.524845 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.524863 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.525934 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.525941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.526264 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.527113 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:52.527141 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:52.529635 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:52.533189 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:52.533198 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:52.533214 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:52.533723 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:52.534106 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:52.534109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:52.534112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:52.534330 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:52.535031 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:52.535042 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:52.535422 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 18:49:52.535680 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 18:49:52.536150 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":358,"threads_started":1}
I20260927 18:49:52.539317 19203 heartbeater.cc:347] Connected to a master server at 127.31.26.254:33165
I20260927 18:49:52.539678 19203 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:52.540403 19203 heartbeater.cc:511] Master 127.31.26.254:33165 requested a full tablet report, sending...
I20260927 18:49:52.541891 18746 ts_manager.cc:194] Registered new tserver with Master: 82df159762b74a47a9779cc858fb14c1 (127.31.26.195:35063)
I20260927 18:49:52.542764 18746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.26.195@KRBTEST.COM'} at 127.31.26.195:42401
I20260927 18:49:52.552246 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:52.570293 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.556931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.31.26.254:33165)
0927 18:49:52.557192 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.557196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.557769 (+ 573us) server_negotiation.cc:407] Connection header received
0927 18:49:52.557934 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.557938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.558007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.558101 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:52.559631 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.560374 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.561825 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.562272 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.562384 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.564823 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:52.564829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:52.564832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:52.564871 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:52.567663 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:52.568363 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:52.568367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:52.568369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:52.568487 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:52.568895 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:52.568898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:52.568900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:52.569324 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:52.569519 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.569890 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.570067 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:49:52.574589 18746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:52.581900 18746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:52.619946 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.595849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.31.26.194:39267)
0927 18:49:52.610095 (+ 14246us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.610110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.610814 (+ 704us) server_negotiation.cc:407] Connection header received
0927 18:49:52.611092 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.611098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.611271 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.611504 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:52.612484 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.613963 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.617133 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.618387 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.618528 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.618764 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.619402 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.619579 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14014,"thread_start_us":13837,"threads_started":1}
I20260927 18:49:52.620052 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.595165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:39267 (local address 127.0.0.1:41930)
0927 18:49:52.610554 (+ 15389us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:52.610570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:52.610898 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:52.611743 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:52.611749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:52.611775 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:52.612278 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.612292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.614133 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.614138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.616935 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.616950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.618889 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.618895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.619118 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.619179 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 18:49:52.619336 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 18:49:52.619559 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15268,"mutex_wait_us":8,"thread_start_us":13385,"threads_started":1}
I20260927 18:49:52.620746 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.595078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:44801 (local address 127.0.0.1:47334)
0927 18:49:52.607917 (+ 12839us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:52.607999 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:52.608262 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:52.611401 (+ 3139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:52.611411 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:52.611449 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:52.611984 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.611997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.615285 (+ 3288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.615290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.617694 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.617709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.619888 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.619895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.620125 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.620171 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:49:52.620263 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:49:52.620402 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12422,"thread_start_us":15203,"threads_started":1}
I20260927 18:49:52.621280 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.595395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.31.26.193:44801)
0927 18:49:52.610573 (+ 15178us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.610585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.610620 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:49:52.610733 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.610742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.611145 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.611367 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:52.612212 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.615123 (+ 2911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.617869 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.619384 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.619509 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.619725 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.620643 (+ 918us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.620836 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14977,"thread_start_us":14798,"threads_started":1}
I20260927 18:49:52.623817 19004 tablet_service.cc:1467] Processing CreateTablet for tablet d3780812ff244cbb8a48f27d04d470f3 (DEFAULT_TABLE table=test-table [id=276a066d0d03468a801317ebae177043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:52.624291 18870 tablet_service.cc:1467] Processing CreateTablet for tablet d3780812ff244cbb8a48f27d04d470f3 (DEFAULT_TABLE table=test-table [id=276a066d0d03468a801317ebae177043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:52.625346 19004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3780812ff244cbb8a48f27d04d470f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:52.625654 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3780812ff244cbb8a48f27d04d470f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:52.628762 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.610754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35063 (local address 127.0.0.1:56116)
0927 18:49:52.615769 (+ 5015us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:52.615786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:52.615939 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:52.616493 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:52.616498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:52.616522 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:52.616970 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.616981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.618943 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.618950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.625521 (+ 6571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.625545 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.627892 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.627903 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.628100 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.628141 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:49:52.628230 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:49:52.628397 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4917,"thread_start_us":4635,"threads_started":1}
I20260927 18:49:52.629642 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.611072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.31.26.195:35063)
0927 18:49:52.615343 (+ 4271us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.615352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.615898 (+ 546us) server_negotiation.cc:407] Connection header received
0927 18:49:52.616122 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.616127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.616301 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.616506 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:52.617199 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.618797 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.625791 (+ 6994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.627331 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.627494 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.627724 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.629111 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.629297 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4092,"thread_start_us":3949,"threads_started":1}
I20260927 18:49:52.634028 19138 tablet_service.cc:1467] Processing CreateTablet for tablet d3780812ff244cbb8a48f27d04d470f3 (DEFAULT_TABLE table=test-table [id=276a066d0d03468a801317ebae177043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:52.635787 19138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3780812ff244cbb8a48f27d04d470f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:52.638963 19222 tablet_bootstrap.cc:492] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Bootstrap starting.
I20260927 18:49:52.642249 19223 tablet_bootstrap.cc:492] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Bootstrap starting.
I20260927 18:49:52.642988 19222 tablet_bootstrap.cc:654] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:52.644457 19222 log.cc:824] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:52.645879 19223 tablet_bootstrap.cc:654] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:52.647140 19223 log.cc:824] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:52.647994 19222 tablet_bootstrap.cc:492] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: No bootstrap required, opened a new log
I20260927 18:49:52.648307 19222 ts_tablet_manager.cc:1397] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260927 18:49:52.649995 19223 tablet_bootstrap.cc:492] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: No bootstrap required, opened a new log
I20260927 18:49:52.650276 19223 ts_tablet_manager.cc:1397] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 18:49:52.652143 19226 tablet_bootstrap.cc:492] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Bootstrap starting.
I20260927 18:49:52.656183 19226 tablet_bootstrap.cc:654] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:52.657588 19226 log.cc:824] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:52.658053 19222 raft_consensus.cc:348] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.658588 19222 raft_consensus.cc:374] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:52.658710 19222 raft_consensus.cc:729] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9da5bbef27460d84838fa77299d1f8, State: Initialized, Role: FOLLOWER
I20260927 18:49:52.659394 19222 consensus_queue.cc:260] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.661257 19226 tablet_bootstrap.cc:492] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: No bootstrap required, opened a new log
I20260927 18:49:52.661327 19222 ts_tablet_manager.cc:1428] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260927 18:49:52.661567 19226 ts_tablet_manager.cc:1397] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 18:49:52.662642 19223 raft_consensus.cc:348] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.663096 19223 raft_consensus.cc:374] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:52.663216 19223 raft_consensus.cc:729] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37108b142fb4046b4b0142a64d90729, State: Initialized, Role: FOLLOWER
I20260927 18:49:52.663959 19223 consensus_queue.cc:260] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.665118 19069 heartbeater.cc:503] Master 127.31.26.254:33165 was elected leader, sending a full tablet report...
I20260927 18:49:52.665798 19223 ts_tablet_manager.cc:1428] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 18:49:52.670418 19226 raft_consensus.cc:348] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.670776 19226 raft_consensus.cc:374] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:52.670851 19226 raft_consensus.cc:729] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82df159762b74a47a9779cc858fb14c1, State: Initialized, Role: FOLLOWER
I20260927 18:49:52.671341 19226 consensus_queue.cc:260] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.672237 19203 heartbeater.cc:503] Master 127.31.26.254:33165 was elected leader, sending a full tablet report...
I20260927 18:49:52.672730 19226 ts_tablet_manager.cc:1428] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260927 18:49:52.685545 18936 tablet.cc:2369] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:52.722834 19070 tablet.cc:2369] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:52.765225 19204 tablet.cc:2369] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:52.904444 19229 raft_consensus.cc:482] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:52.904786 19229 raft_consensus.cc:504] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.906600 19229 leader_election.cc:288] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9da5bbef27460d84838fa77299d1f8 (127.31.26.193:44801), 82df159762b74a47a9779cc858fb14c1 (127.31.26.195:35063)
I20260927 18:49:52.913342 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.907049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:59425 (local address 127.31.26.193:44801)
0927 18:49:52.907247 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.907252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.907698 (+ 446us) server_negotiation.cc:407] Connection header received
0927 18:49:52.907921 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.907926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.908047 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.908185 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:52.908755 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.909585 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.911419 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.912303 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.912493 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.912576 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.912874 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.913034 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:49:52.913439 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.906900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:44801 (local address 127.31.26.194:59425)
0927 18:49:52.907573 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:52.907592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:52.907756 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:52.908216 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:52.908221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:52.908255 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:52.908600 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.908610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.909715 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.909723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.911257 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.911269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.912536 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.912542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.912700 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.912725 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:49:52.912830 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:49:52.913017 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":334,"threads_started":1}
I20260927 18:49:52.914178 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.907939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:59873 (local address 127.31.26.195:35063)
0927 18:49:52.908429 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.908438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.908540 (+ 102us) server_negotiation.cc:407] Connection header received
0927 18:49:52.908749 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.908753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.908811 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.908911 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:52.909524 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.910263 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.912102 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.912946 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.913162 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.913237 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.913795 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.913939 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260927 18:49:52.914223 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.907693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:35063 (local address 127.31.26.194:59873)
0927 18:49:52.908418 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:52.908440 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:52.908589 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:52.908940 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:52.908944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:52.908969 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:52.909376 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.909384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.910401 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.910408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.911959 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:52.911972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.913196 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.913202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:52.913379 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.913405 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:49:52.913734 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 18:49:52.913865 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":355,"threads_started":1}
I20260927 18:49:52.914685 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3780812ff244cbb8a48f27d04d470f3" candidate_uuid: "a37108b142fb4046b4b0142a64d90729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9da5bbef27460d84838fa77299d1f8" is_pre_election: true
I20260927 18:49:52.915305 18890 raft_consensus.cc:2454] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a37108b142fb4046b4b0142a64d90729 in term 0.
I20260927 18:49:52.915355 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3780812ff244cbb8a48f27d04d470f3" candidate_uuid: "a37108b142fb4046b4b0142a64d90729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82df159762b74a47a9779cc858fb14c1" is_pre_election: true
I20260927 18:49:52.915881 19158 raft_consensus.cc:2454] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a37108b142fb4046b4b0142a64d90729 in term 0.
I20260927 18:49:52.915997 18960 leader_election.cc:302] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a37108b142fb4046b4b0142a64d90729, ba9da5bbef27460d84838fa77299d1f8; no voters:
I20260927 18:49:52.916445 19229 raft_consensus.cc:2781] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:52.916607 19229 raft_consensus.cc:482] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:52.916733 19229 raft_consensus.cc:3037] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:52.919016 19229 raft_consensus.cc:504] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.919651 19229 leader_election.cc:288] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [CANDIDATE]: Term 1 election: Requested vote from peers ba9da5bbef27460d84838fa77299d1f8 (127.31.26.193:44801), 82df159762b74a47a9779cc858fb14c1 (127.31.26.195:35063)
I20260927 18:49:52.920197 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3780812ff244cbb8a48f27d04d470f3" candidate_uuid: "a37108b142fb4046b4b0142a64d90729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9da5bbef27460d84838fa77299d1f8"
I20260927 18:49:52.920252 19158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3780812ff244cbb8a48f27d04d470f3" candidate_uuid: "a37108b142fb4046b4b0142a64d90729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82df159762b74a47a9779cc858fb14c1"
I20260927 18:49:52.920420 18890 raft_consensus.cc:3037] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:52.920480 19158 raft_consensus.cc:3037] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:52.922869 18890 raft_consensus.cc:2454] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a37108b142fb4046b4b0142a64d90729 in term 1.
I20260927 18:49:52.922873 19158 raft_consensus.cc:2454] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a37108b142fb4046b4b0142a64d90729 in term 1.
I20260927 18:49:52.923344 18960 leader_election.cc:302] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a37108b142fb4046b4b0142a64d90729, ba9da5bbef27460d84838fa77299d1f8; no voters:
I20260927 18:49:52.923677 19229 raft_consensus.cc:2781] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:52.924139 19229 raft_consensus.cc:686] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 1 LEADER]: Becoming Leader. State: Replica: a37108b142fb4046b4b0142a64d90729, State: Running, Role: LEADER
I20260927 18:49:52.924647 19229 consensus_queue.cc:237] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } }
I20260927 18:49:52.929863 18746 catalog_manager.cc:5495] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 reported cstate change: term changed from 0 to 1, leader changed from <none> to a37108b142fb4046b4b0142a64d90729 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "a37108b142fb4046b4b0142a64d90729" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a37108b142fb4046b4b0142a64d90729" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 39267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 } health_report { overall_health: UNKNOWN } } }
I20260927 18:49:52.977562 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:52.971355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.31.26.194:39267)
0927 18:49:52.971571 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:52.971576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:52.971593 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:49:52.971759 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:52.971763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:52.971836 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:52.971955 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:52.972405 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.973206 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.975194 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:52.975694 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:52.975790 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:52.976311 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:52.976848 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:52.976955 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:52.977161 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:49:52.977232 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:49:52.986290 18890 raft_consensus.cc:1261] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 1 FOLLOWER]: Refusing update from remote peer a37108b142fb4046b4b0142a64d90729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:52.986297 19158 raft_consensus.cc:1261] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 1 FOLLOWER]: Refusing update from remote peer a37108b142fb4046b4b0142a64d90729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:52.987346 19229 consensus_queue.cc:1035] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9da5bbef27460d84838fa77299d1f8" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:52.987751 19233 consensus_queue.cc:1035] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82df159762b74a47a9779cc858fb14c1" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:52.998471 19236 mvcc.cc:205] Tried to move back new op lower bound from 7334031331260493824 to 7334031331024175104. Current Snapshot: MvccSnapshot[applied={T|T < 7334031331260493824}]
I20260927 18:49:52.999578 19237 mvcc.cc:205] Tried to move back new op lower bound from 7334031331260493824 to 7334031331024175104. Current Snapshot: MvccSnapshot[applied={T|T < 7334031331260493824}]
I20260927 18:49:53.018199 18745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:53.018504 18745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:53.022697 18745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67952e48b1d249a298fc4e53a68cffe9: Sending DeleteTablet for 3 replicas of tablet d3780812ff244cbb8a48f27d04d470f3
I20260927 18:49:53.024044 19004 tablet_service.cc:1514] Processing DeleteTablet for tablet d3780812ff244cbb8a48f27d04d470f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:53 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:41930
I20260927 18:49:53.024044 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet d3780812ff244cbb8a48f27d04d470f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:53 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:47334
I20260927 18:49:53.024278 19138 tablet_service.cc:1514] Processing DeleteTablet for tablet d3780812ff244cbb8a48f27d04d470f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:53 UTC) from {username='kudu', principal='kudu/127.31.26.254@KRBTEST.COM'} at 127.0.0.1:56116
I20260927 18:49:53.025899 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18807
I20260927 18:49:53.036682 19246 tablet_replica.cc:330] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: stopping tablet replica
I20260927 18:49:53.037731 19246 raft_consensus.cc:2229] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:53.039062 19246 raft_consensus.cc:2258] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:53.039824 19248 tablet_replica.cc:330] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: stopping tablet replica
I20260927 18:49:53.040304 19248 raft_consensus.cc:2229] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:53.040817 19248 raft_consensus.cc:2258] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:53.041155 19247 tablet_replica.cc:330] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: stopping tablet replica
I20260927 18:49:53.042666 19246 ts_tablet_manager.cc:1905] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:53.043228 19247 raft_consensus.cc:2229] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:53.043417 19248 ts_tablet_manager.cc:1905] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:53.043891 19247 raft_consensus.cc:2258] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:53.051084 19247 ts_tablet_manager.cc:1905] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:53.051415 19248 ts_tablet_manager.cc:1918] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:53.051594 19248 log.cc:1196] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/d3780812ff244cbb8a48f27d04d470f3
I20260927 18:49:53.052134 19248 ts_tablet_manager.cc:1939] T d3780812ff244cbb8a48f27d04d470f3 P 82df159762b74a47a9779cc858fb14c1: Deleting consensus metadata
I20260927 18:49:53.054005 19246 ts_tablet_manager.cc:1918] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:53.054355 19246 log.cc:1196] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/d3780812ff244cbb8a48f27d04d470f3
I20260927 18:49:53.054350 18733 catalog_manager.cc:4841] TS 82df159762b74a47a9779cc858fb14c1 (127.31.26.195:35063): tablet d3780812ff244cbb8a48f27d04d470f3 (table test-table [id=276a066d0d03468a801317ebae177043]) successfully deleted
I20260927 18:49:53.055114 19246 ts_tablet_manager.cc:1939] T d3780812ff244cbb8a48f27d04d470f3 P a37108b142fb4046b4b0142a64d90729: Deleting consensus metadata
Sep 27 18:49:53 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.193@KRBTEST.COM
I20260927 18:49:53.057288 18730 catalog_manager.cc:4841] TS a37108b142fb4046b4b0142a64d90729 (127.31.26.194:39267): tablet d3780812ff244cbb8a48f27d04d470f3 (table test-table [id=276a066d0d03468a801317ebae177043]) successfully deleted
I20260927 18:49:53.061368 19247 ts_tablet_manager.cc:1918] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:53.061563 19247 log.cc:1196] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534668899567-31851-0/minicluster-data/ts-0/wal/wals/d3780812ff244cbb8a48f27d04d470f3
I20260927 18:49:53.062114 19247 ts_tablet_manager.cc:1939] T d3780812ff244cbb8a48f27d04d470f3 P ba9da5bbef27460d84838fa77299d1f8: Deleting consensus metadata
I20260927 18:49:53.064131 18733 catalog_manager.cc:4841] TS ba9da5bbef27460d84838fa77299d1f8 (127.31.26.193:44801): tablet d3780812ff244cbb8a48f27d04d470f3 (table test-table [id=276a066d0d03468a801317ebae177043]) successfully deleted
I20260927 18:49:53.064359 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:53.027838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.31.26.193:44801)
0927 18:49:53.028045 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:53.028052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:53.041805 (+ 13753us) server_negotiation.cc:407] Connection header received
0927 18:49:53.042138 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:53.042144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:53.042236 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:53.042314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:53.044240 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:53.045420 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:53.047842 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:53.048545 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:53.048657 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:53.057709 (+ 9052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:53.057731 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:53.057754 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:53.057808 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:53.061134 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:53.062069 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:53.062081 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:53.062089 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:53.062311 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:53.062731 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:53.062736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:53.062739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:53.063130 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:53.063459 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:53.063757 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:49:53.063980 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:49:53.065155 18930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:53.156236 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18807
I20260927 18:49:53.174532 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18941
Sep 27 18:49:53 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.194@KRBTEST.COM
I20260927 18:49:53.190925 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:53.175362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.31.26.194:39267)
0927 18:49:53.175537 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:53.175545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:53.175615 (+ 70us) server_negotiation.cc:407] Connection header received
0927 18:49:53.175813 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:53.175819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:53.175897 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:53.176033 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:53.177330 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:53.178138 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:53.179459 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:53.179908 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:53.180077 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:53.184694 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:53.184716 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:53.184733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:53.184775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:53.187915 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:53.188675 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:53.188684 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:53.188687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:53.188839 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:53.189301 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:53.189306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:53.189311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:53.189668 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:53.190048 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:53.190323 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 18:49:53.190556 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:49:53.191588 19064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:53.308385 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18941
I20260927 18:49:53.327649 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19075
Sep 27 18:49:53 dist-test-slave-rfd5 krb5kdc[18701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.195@KRBTEST.COM
I20260927 18:49:53.344238 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:53.328494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.31.26.195:35063)
0927 18:49:53.328716 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:53.328722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:53.328770 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:49:53.328998 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:53.329002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:53.329073 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:53.329236 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:53.330677 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:53.331453 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:53.332806 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:53.333301 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:53.333438 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:53.338150 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:53.338168 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:53.338185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:53.338226 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:53.341073 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:53.341990 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:53.342000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:53.342004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:53.342157 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:53.342562 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:53.342568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:53.342571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:53.342995 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:53.343306 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:53.343656 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:49:53.343885 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 18:49:53.345000 19198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:53.434798 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19075
I20260927 18:49:53.451699 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 18714
I20260927 18:49:53.452889 18776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:53.565819 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 18714
2026-09-27T18:49:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4633 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T18:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:53Z Disabled control of system clock
I20260927 18:49:53.644636 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:32991
--webserver_interface=127.31.26.254
--webserver_port=0
--builtin_ntp_servers=127.31.26.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:49:53.991575 19257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:53.991894 19257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:53.992048 19257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:53.999684 19257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:53.999804 19257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:53.999876 19257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:53.999979 19257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:54.000063 19257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:54.009836 19257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:32991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:54.011219 19257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:54.013131 19257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:54.025647 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:54.025563 19265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:54.025625 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:54.026646 19257 server_base.cc:1034] running on GCE node
I20260927 18:49:54.027475 19257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:54.028980 19257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:54.030220 19257 hybrid_clock.cc:648] HybridClock initialized: now 1790534994030169 us; error 37 us; skew 500 ppm
I20260927 18:49:54.030573 19257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:54.033305 19257 webserver.cc:460] Webserver started at http://127.31.26.254:39861/ using document root <none> and password file <none>
I20260927 18:49:54.034034 19257 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:54.034127 19257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:54.034399 19257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:54.037500 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1"
format_stamp: "Formatted at 2026-09-27 18:49:54 on dist-test-slave-rfd5"
server_key: "2fc0a907d0df1e2740947026b19e9212"
server_key_iv: "02f1364c833ee118ed87ebfdce3667a2"
server_key_version: "encryptionkey@0"
I20260927 18:49:54.038318 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1"
format_stamp: "Formatted at 2026-09-27 18:49:54 on dist-test-slave-rfd5"
server_key: "2fc0a907d0df1e2740947026b19e9212"
server_key_iv: "02f1364c833ee118ed87ebfdce3667a2"
server_key_version: "encryptionkey@0"
I20260927 18:49:54.043891 19257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:49:54.047756 19271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:54.049425 19257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:49:54.049598 19257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1"
format_stamp: "Formatted at 2026-09-27 18:49:54 on dist-test-slave-rfd5"
server_key: "2fc0a907d0df1e2740947026b19e9212"
server_key_iv: "02f1364c833ee118ed87ebfdce3667a2"
server_key_version: "encryptionkey@0"
I20260927 18:49:54.049840 19257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:54.064075 19257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:54.065002 19257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:54.065354 19257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:54.085225 19257 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:32991
I20260927 18:49:54.085258 19322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:32991 every 8 connection(s)
I20260927 18:49:54.086735 19257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:54.090786 19323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:54.090993 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19257
I20260927 18:49:54.091190 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:54.091663 31851 external_mini_cluster.cc:1428] Setting key 05ea832dfaf5340d6abe5a0c9bb4b838
I20260927 18:49:54.106148 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1: Bootstrap starting.
I20260927 18:49:54.108687 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:54.093781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.31.26.254:32991)
0927 18:49:54.094997 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:54.095012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:54.095067 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:49:54.097882 (+ 2815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:54.097915 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:54.098629 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:54.099150 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:54.099683 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:54.101609 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:54.103041 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:54.104156 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:54.104334 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:54.105632 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:54.105648 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:54.105669 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:54.106436 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:54.106554 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:54.106584 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:54.106701 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:54.106930 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:49:54.107292 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":369,"threads_started":1}
I20260927 18:49:54.110474 19323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:54.112164 19323 log.cc:824] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:54.116151 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1: No bootstrap required, opened a new log
I20260927 18:49:54.124207 19323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 32991 } }
I20260927 18:49:54.124670 19323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:54.124774 19323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942dd7eadb5a4e7695b5a70d4f8e52f1, State: Initialized, Role: FOLLOWER
I20260927 18:49:54.125548 19323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 32991 } }
I20260927 18:49:54.125825 19323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:54.125985 19323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:54.126181 19323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:54.128438 19323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 32991 } }
I20260927 18:49:54.128996 19323 leader_election.cc:302] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942dd7eadb5a4e7695b5a70d4f8e52f1; no voters:
I20260927 18:49:54.129649 19323 leader_election.cc:288] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:54.129837 19328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:54.131254 19328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [term 1 LEADER]: Becoming Leader. State: Replica: 942dd7eadb5a4e7695b5a70d4f8e52f1, State: Running, Role: LEADER
I20260927 18:49:54.131839 19328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 32991 } }
I20260927 18:49:54.131969 19323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:54.135059 19330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942dd7eadb5a4e7695b5a70d4f8e52f1. Latest consensus state: current_term: 1 leader_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 32991 } } }
I20260927 18:49:54.135207 19329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942dd7eadb5a4e7695b5a70d4f8e52f1" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 32991 } } }
I20260927 18:49:54.135604 19330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:54.135614 19329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:54.137480 19336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:54.144080 19336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:54.154424 19336 catalog_manager.cc:1347] Generated new cluster ID: 59cb8c06d8f745ee9d40f6917db8bd47
I20260927 18:49:54.154589 19336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:54.175184 19336 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:54.176579 19336 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:54.187997 19336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1: Generated new TSK 0
I20260927 18:49:54.189234 19336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:49:54.200294 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.193:0
--local_ip_for_outbound_sockets=127.31.26.193
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:32991
--builtin_ntp_servers=127.31.26.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:49:54.560564 19347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:54.560891 19347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:54.560997 19347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:54.568800 19347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:54.568931 19347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:54.569183 19347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.193
I20260927 18:49:54.579169 19347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.26.193
--webserver_port=0
--tserver_master_addrs=127.31.26.254:32991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.193
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:54.580572 19347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:54.582535 19347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:54.597201 19352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:54.597191 19355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:54.597196 19353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:54.598052 19347 server_base.cc:1034] running on GCE node
I20260927 18:49:54.598693 19347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:54.599704 19347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:54.600925 19347 hybrid_clock.cc:648] HybridClock initialized: now 1790534994600854 us; error 70 us; skew 500 ppm
I20260927 18:49:54.601357 19347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:54.603966 19347 webserver.cc:460] Webserver started at http://127.31.26.193:35567/ using document root <none> and password file <none>
I20260927 18:49:54.604736 19347 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:54.604859 19347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:54.605319 19347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:54.608381 19347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data/instance:
uuid: "4c804978c3bc42a4ba9521605a21bab9"
format_stamp: "Formatted at 2026-09-27 18:49:54 on dist-test-slave-rfd5"
server_key: "674f9009f8db0036a240906f0ec9fe35"
server_key_iv: "ac41fd524a3ced1b5c25806ae9259437"
server_key_version: "encryptionkey@0"
I20260927 18:49:54.609320 19347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance:
uuid: "4c804978c3bc42a4ba9521605a21bab9"
format_stamp: "Formatted at 2026-09-27 18:49:54 on dist-test-slave-rfd5"
server_key: "674f9009f8db0036a240906f0ec9fe35"
server_key_iv: "ac41fd524a3ced1b5c25806ae9259437"
server_key_version: "encryptionkey@0"
I20260927 18:49:54.614833 19347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:49:54.618593 19361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:54.620039 19347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:49:54.620250 19347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal
uuid: "4c804978c3bc42a4ba9521605a21bab9"
format_stamp: "Formatted at 2026-09-27 18:49:54 on dist-test-slave-rfd5"
server_key: "674f9009f8db0036a240906f0ec9fe35"
server_key_iv: "ac41fd524a3ced1b5c25806ae9259437"
server_key_version: "encryptionkey@0"
I20260927 18:49:54.620489 19347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:54.637259 19347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:54.638236 19347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:54.638550 19347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:54.639568 19347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:54.641147 19347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:54.641283 19347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:54.641422 19347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:54.641508 19347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:54.676204 19347 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.193:33349
I20260927 18:49:54.676266 19473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.193:33349 every 8 connection(s)
I20260927 18:49:54.677788 19347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:54.686961 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19347
I20260927 18:49:54.687227 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:54.687685 31851 external_mini_cluster.cc:1428] Setting key 4d65ba23d2f12a1c886aba4524e3d41f
I20260927 18:49:54.691009 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.194:0
--local_ip_for_outbound_sockets=127.31.26.194
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:32991
--builtin_ntp_servers=127.31.26.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:49:54.691421 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:54.680503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.193:48669 (local address 127.31.26.254:32991)
0927 18:49:54.680959 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:54.680964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:54.682313 (+ 1349us) server_negotiation.cc:407] Connection header received
0927 18:49:54.684150 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:54.684155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:54.684227 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:54.684335 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:54.685609 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:54.686336 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:54.688251 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:54.688695 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:54.688820 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:54.690441 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:54.690446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:54.690450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:54.690468 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:54.690509 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:54.690521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:54.690588 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:54.690909 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 18:49:54.691036 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260927 18:49:54.692893 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:54.681212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:32991 (local address 127.31.26.193:48669)
0927 18:49:54.682159 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:54.682197 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:54.683860 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:54.684599 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:54.684608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:54.684726 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:54.685433 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:54.685454 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:54.686545 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:54.686550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:54.688102 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:54.688115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:54.688923 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:54.688929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:54.689205 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:54.690032 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:54.690036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:54.690095 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:54.690224 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:54.690624 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:54.690637 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:54.690646 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:49:54.690850 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 18:49:54.691336 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":295,"threads_started":1}
I20260927 18:49:54.694581 19474 heartbeater.cc:347] Connected to a master server at 127.31.26.254:32991
I20260927 18:49:54.694938 19474 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:54.695741 19474 heartbeater.cc:511] Master 127.31.26.254:32991 requested a full tablet report, sending...
I20260927 18:49:54.698101 19288 ts_manager.cc:194] Registered new tserver with Master: 4c804978c3bc42a4ba9521605a21bab9 (127.31.26.193:33349)
I20260927 18:49:54.699651 19288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.193:48669
W20260927 18:49:55.032568 19478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:55.032938 19478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:55.033164 19478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:55.040903 19478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:55.041074 19478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:55.041288 19478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.194
I20260927 18:49:55.050819 19478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.26.194
--webserver_port=0
--tserver_master_addrs=127.31.26.254:32991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.194
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:55.052197 19478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:55.054243 19478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:55.067919 19484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:55.068066 19486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:55.068025 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:55.068903 19478 server_base.cc:1034] running on GCE node
I20260927 18:49:55.069620 19478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:55.070780 19478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:55.072074 19478 hybrid_clock.cc:648] HybridClock initialized: now 1790534995072016 us; error 64 us; skew 500 ppm
I20260927 18:49:55.072464 19478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:55.074945 19478 webserver.cc:460] Webserver started at http://127.31.26.194:37737/ using document root <none> and password file <none>
I20260927 18:49:55.075668 19478 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:55.075803 19478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:55.076118 19478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:55.079092 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data/instance:
uuid: "66034359f4e14449810e6d4f55d7b780"
format_stamp: "Formatted at 2026-09-27 18:49:55 on dist-test-slave-rfd5"
server_key: "aad974400b435627e104ceabde260945"
server_key_iv: "89325b75546a2357e74f46c4321384ea"
server_key_version: "encryptionkey@0"
I20260927 18:49:55.079888 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance:
uuid: "66034359f4e14449810e6d4f55d7b780"
format_stamp: "Formatted at 2026-09-27 18:49:55 on dist-test-slave-rfd5"
server_key: "aad974400b435627e104ceabde260945"
server_key_iv: "89325b75546a2357e74f46c4321384ea"
server_key_version: "encryptionkey@0"
I20260927 18:49:55.085445 19478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 18:49:55.089869 19492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:55.091442 19478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:55.091598 19478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal
uuid: "66034359f4e14449810e6d4f55d7b780"
format_stamp: "Formatted at 2026-09-27 18:49:55 on dist-test-slave-rfd5"
server_key: "aad974400b435627e104ceabde260945"
server_key_iv: "89325b75546a2357e74f46c4321384ea"
server_key_version: "encryptionkey@0"
I20260927 18:49:55.091818 19478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:55.114598 19478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:55.115459 19478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:55.115728 19478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:55.116729 19478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:55.118377 19478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:55.118471 19478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:55.118580 19478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:55.118672 19478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:55.152704 19478 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.194:42173
I20260927 18:49:55.152757 19604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.194:42173 every 8 connection(s)
I20260927 18:49:55.154325 19478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:55.159554 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19478
I20260927 18:49:55.159754 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:55.160197 31851 external_mini_cluster.cc:1428] Setting key 80f35e6a21697c0dcb2ee481f40c236f
I20260927 18:49:55.162978 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.26.195:0
--local_ip_for_outbound_sockets=127.31.26.195
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:32991
--builtin_ntp_servers=127.31.26.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:49:55.167311 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.156981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:55205 (local address 127.31.26.254:32991)
0927 18:49:55.157192 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.157198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.158668 (+ 1470us) server_negotiation.cc:407] Connection header received
0927 18:49:55.160626 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.160632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.160725 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.160857 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:55.162005 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.162712 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.164366 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.164780 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.164919 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.166443 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:55.166449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:55.166460 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:55.166482 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:55.166518 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:55.166530 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:55.166631 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.166872 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.166981 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:49:55.168411 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.157607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:32991 (local address 127.31.26.194:55205)
0927 18:49:55.158519 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.158554 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.160342 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.161150 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.161159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.161254 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:55.161832 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.161849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.162864 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.162872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.164219 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.164231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.165012 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.165019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.165323 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.166089 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:55.166091 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:55.166140 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:55.166229 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:55.166630 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:55.166645 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:55.166655 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.166846 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.167297 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260927 18:49:55.170045 19605 heartbeater.cc:347] Connected to a master server at 127.31.26.254:32991
I20260927 18:49:55.170378 19605 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:55.171177 19605 heartbeater.cc:511] Master 127.31.26.254:32991 requested a full tablet report, sending...
I20260927 18:49:55.172791 19288 ts_manager.cc:194] Registered new tserver with Master: 66034359f4e14449810e6d4f55d7b780 (127.31.26.194:42173)
I20260927 18:49:55.174100 19288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.194:55205
W20260927 18:49:55.511016 19608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:55.511335 19608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:55.511432 19608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:55.519820 19608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:55.519963 19608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:55.520081 19608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.26.195
I20260927 18:49:55.530220 19608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.26.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.26.195
--webserver_port=0
--tserver_master_addrs=127.31.26.254:32991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.26.195
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:55.531528 19608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:55.533630 19608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:55.547767 19613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:55.547717 19616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:55.547737 19614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:55.548614 19608 server_base.cc:1034] running on GCE node
I20260927 18:49:55.549374 19608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:55.550371 19608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:55.551623 19608 hybrid_clock.cc:648] HybridClock initialized: now 1790534995551578 us; error 37 us; skew 500 ppm
I20260927 18:49:55.552057 19608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:49:55.554795 19608 webserver.cc:460] Webserver started at http://127.31.26.195:42821/ using document root <none> and password file <none>
I20260927 18:49:55.555558 19608 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:55.555655 19608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:55.556033 19608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:55.559319 19608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data/instance:
uuid: "0b15e31407004141abfdcd826eed33e2"
format_stamp: "Formatted at 2026-09-27 18:49:55 on dist-test-slave-rfd5"
server_key: "77d00ec0673ae782a3f07401f49ea915"
server_key_iv: "35ca648797d92afc7c26812db34767e8"
server_key_version: "encryptionkey@0"
I20260927 18:49:55.560210 19608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance:
uuid: "0b15e31407004141abfdcd826eed33e2"
format_stamp: "Formatted at 2026-09-27 18:49:55 on dist-test-slave-rfd5"
server_key: "77d00ec0673ae782a3f07401f49ea915"
server_key_iv: "35ca648797d92afc7c26812db34767e8"
server_key_version: "encryptionkey@0"
I20260927 18:49:55.565836 19608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:49:55.569743 19622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:55.571257 19608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:49:55.571440 19608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal
uuid: "0b15e31407004141abfdcd826eed33e2"
format_stamp: "Formatted at 2026-09-27 18:49:55 on dist-test-slave-rfd5"
server_key: "77d00ec0673ae782a3f07401f49ea915"
server_key_iv: "35ca648797d92afc7c26812db34767e8"
server_key_version: "encryptionkey@0"
I20260927 18:49:55.571687 19608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:55.601845 19608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:55.602756 19608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:55.603078 19608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:55.604244 19608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:55.605896 19608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:55.606031 19608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:55.606177 19608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:55.606268 19608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:55.640777 19608 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.195:37033
I20260927 18:49:55.640872 19734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.195:37033 every 8 connection(s)
I20260927 18:49:55.642184 19608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:55.650415 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19608
I20260927 18:49:55.650640 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:55.651268 31851 external_mini_cluster.cc:1428] Setting key 5dfa24ea4d10cda889da5e2bdeb4833f
I20260927 18:49:55.655846 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.644761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.195:51627 (local address 127.31.26.254:32991)
0927 18:49:55.644939 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.644943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.646462 (+ 1519us) server_negotiation.cc:407] Connection header received
0927 18:49:55.648309 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.648314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.648411 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.648523 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:55.649739 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.650509 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.652650 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.653296 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.653457 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.654997 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:55.655004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:55.655008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:55.655035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:55.655068 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:55.655080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:55.655188 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.655419 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.655518 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:49:55.656972 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.645327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:32991 (local address 127.31.26.195:51627)
0927 18:49:55.646274 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.646316 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.648015 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.648781 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.648790 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.648889 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:55.649550 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.649571 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.650730 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.650736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.652504 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.652518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.653552 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.653559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.653808 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.654669 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:49:55.654671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:55.654718 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:49:55.654800 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:55.655186 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:55.655199 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:55.655210 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.655414 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.655844 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":301,"threads_started":1}
I20260927 18:49:55.658401 19735 heartbeater.cc:347] Connected to a master server at 127.31.26.254:32991
I20260927 18:49:55.658711 19735 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:55.659482 19735 heartbeater.cc:511] Master 127.31.26.254:32991 requested a full tablet report, sending...
I20260927 18:49:55.660943 19288 ts_manager.cc:194] Registered new tserver with Master: 0b15e31407004141abfdcd826eed33e2 (127.31.26.195:37033)
I20260927 18:49:55.661842 19288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.26.195:51627
I20260927 18:49:55.666801 31851 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:55.677395 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.670837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.31.26.254:32991)
0927 18:49:55.671092 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.671101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.671451 (+ 350us) server_negotiation.cc:407] Connection header received
0927 18:49:55.671671 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.671675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.671749 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.671855 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:55.672338 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.673066 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.674476 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.674917 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.675052 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.676239 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:55.676244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:55.676248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:55.676271 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:55.676313 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:55.676330 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:55.676412 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.676701 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.677054 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:49:55.681663 19288 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:55.688633 19288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:55.703753 19474 heartbeater.cc:503] Master 127.31.26.254:32991 was elected leader, sending a full tablet report...
I20260927 18:49:55.713115 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.701508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:37033 (local address 127.0.0.1:37182)
0927 18:49:55.702544 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.702602 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.702811 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.704814 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.704823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.704854 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:55.705421 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.705438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.707916 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.707921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.710230 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.710242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.711935 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.711942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.712173 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.712231 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.712375 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.712603 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":37,"thread_start_us":1720,"threads_started":1}
I20260927 18:49:55.714249 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.703646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40110)
0927 18:49:55.704681 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.704702 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.704814 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.706015 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.706020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.706049 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:55.706583 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.706598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.708160 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.708166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.710832 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.710849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.713112 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.713121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.713622 (+ 501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.713656 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.713750 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.713881 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":49,"thread_start_us":754,"threads_started":1}
I20260927 18:49:55.714323 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.701771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.31.26.195:37033)
0927 18:49:55.704072 (+ 2301us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.704085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.704158 (+ 73us) server_negotiation.cc:407] Connection header received
0927 18:49:55.704262 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.704270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.704542 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.705674 (+ 1132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:55.706024 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.707768 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.710441 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.711720 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.711873 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.712167 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.712470 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.713974 (+ 1504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":2306,"threads_started":1}
I20260927 18:49:55.714538 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.702171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.31.26.194:42173)
0927 18:49:55.703444 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.703455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.703622 (+ 167us) server_negotiation.cc:407] Connection header received
0927 18:49:55.704007 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.704016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.705640 (+ 1624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.705833 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:55.706899 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.708357 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.711260 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.712594 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.712738 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.713001 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.714010 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.714181 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":1007,"threads_started":1}
I20260927 18:49:55.714740 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.701656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.194:42173 (local address 127.0.0.1:55888)
0927 18:49:55.703513 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.703532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.703730 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.706110 (+ 2380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.706114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.706176 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:55.706679 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.706694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.708577 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.708585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.711061 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.711074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.713178 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.713186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.713402 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.713451 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.713552 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.713704 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"mutex_wait_us":86,"spinlock_wait_cycles":1408,"thread_start_us":634,"threads_started":1}
I20260927 18:49:55.714397 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.704237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.31.26.193:33349)
0927 18:49:55.705370 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.705381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.705424 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:49:55.705536 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.705547 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.705774 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.706671 (+ 897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:55.706905 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.708006 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.711074 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.712513 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.712643 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.712941 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.713828 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.714011 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":773,"threads_started":1}
I20260927 18:49:55.718462 19540 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb9c5cfe2841d89db7e3f2aade9aae (DEFAULT_TABLE table=test-table [id=c5b40122bf6b415cad9fc4dbe3724f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:55.718502 19670 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb9c5cfe2841d89db7e3f2aade9aae (DEFAULT_TABLE table=test-table [id=c5b40122bf6b415cad9fc4dbe3724f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:55.718502 19409 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb9c5cfe2841d89db7e3f2aade9aae (DEFAULT_TABLE table=test-table [id=c5b40122bf6b415cad9fc4dbe3724f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:55.719781 19540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eb9c5cfe2841d89db7e3f2aade9aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:55.719854 19409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eb9c5cfe2841d89db7e3f2aade9aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:55.719877 19670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79eb9c5cfe2841d89db7e3f2aade9aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:55.733522 19753 tablet_bootstrap.cc:492] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Bootstrap starting.
I20260927 18:49:55.737356 19755 tablet_bootstrap.cc:492] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: Bootstrap starting.
I20260927 18:49:55.737720 19753 tablet_bootstrap.cc:654] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:55.739543 19754 tablet_bootstrap.cc:492] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Bootstrap starting.
I20260927 18:49:55.740129 19753 log.cc:824] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:55.743273 19755 tablet_bootstrap.cc:654] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:55.743520 19754 tablet_bootstrap.cc:654] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:55.744565 19754 log.cc:824] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:55.744755 19755 log.cc:824] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:55.745204 19753 tablet_bootstrap.cc:492] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: No bootstrap required, opened a new log
I20260927 18:49:55.745616 19753 ts_tablet_manager.cc:1397] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260927 18:49:55.747829 19755 tablet_bootstrap.cc:492] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: No bootstrap required, opened a new log
I20260927 18:49:55.748122 19755 ts_tablet_manager.cc:1397] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 18:49:55.748229 19754 tablet_bootstrap.cc:492] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: No bootstrap required, opened a new log
I20260927 18:49:55.748517 19754 ts_tablet_manager.cc:1397] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 18:49:55.755852 19755 raft_consensus.cc:348] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.756286 19755 raft_consensus.cc:374] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:55.756382 19755 raft_consensus.cc:729] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c804978c3bc42a4ba9521605a21bab9, State: Initialized, Role: FOLLOWER
I20260927 18:49:55.757071 19755 consensus_queue.cc:260] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.757051 19753 raft_consensus.cc:348] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.757579 19753 raft_consensus.cc:374] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:55.757694 19753 raft_consensus.cc:729] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66034359f4e14449810e6d4f55d7b780, State: Initialized, Role: FOLLOWER
I20260927 18:49:55.758325 19753 consensus_queue.cc:260] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.760027 19754 raft_consensus.cc:348] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.760673 19754 raft_consensus.cc:374] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:55.760802 19754 raft_consensus.cc:729] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b15e31407004141abfdcd826eed33e2, State: Initialized, Role: FOLLOWER
I20260927 18:49:55.761525 19754 consensus_queue.cc:260] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.761749 19605 heartbeater.cc:503] Master 127.31.26.254:32991 was elected leader, sending a full tablet report...
I20260927 18:49:55.762651 19753 ts_tablet_manager.cc:1428] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260927 18:49:55.764762 19755 ts_tablet_manager.cc:1428] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260927 18:49:55.765807 19735 heartbeater.cc:503] Master 127.31.26.254:32991 was elected leader, sending a full tablet report...
I20260927 18:49:55.767083 19754 ts_tablet_manager.cc:1428] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Time spent starting tablet: real 0.018s user 0.019s sys 0.001s
I20260927 18:49:55.807039 19760 raft_consensus.cc:482] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:55.807461 19760 raft_consensus.cc:504] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.809520 19760 leader_election.cc:288] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b15e31407004141abfdcd826eed33e2 (127.31.26.195:37033), 4c804978c3bc42a4ba9521605a21bab9 (127.31.26.193:33349)
I20260927 18:49:55.816887 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.809945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:54071 (local address 127.31.26.195:37033)
0927 18:49:55.810169 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.810178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.810763 (+ 585us) server_negotiation.cc:407] Connection header received
0927 18:49:55.811002 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.811010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.811101 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.811204 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:55.811876 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.812834 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.814942 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.815825 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.815955 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.816009 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.816442 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.816603 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:49:55.816892 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.809893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.195:37033 (local address 127.31.26.194:54071)
0927 18:49:55.810626 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.810643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.810844 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.811247 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.811252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.811286 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:55.811694 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.811706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.812978 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.812984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.814745 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.814758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.816034 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.816041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.816237 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.816275 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.816382 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.816504 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":77,"thread_start_us":398,"threads_started":1}
I20260927 18:49:55.817181 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.810890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.26.194:44839 (local address 127.31.26.193:33349)
0927 18:49:55.811145 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.811155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.811552 (+ 397us) server_negotiation.cc:407] Connection header received
0927 18:49:55.811666 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.811674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.811774 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.811874 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:55.812455 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.813330 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.815211 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.816069 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.816252 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.816314 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.816744 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.816896 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 18:49:55.817761 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.810710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.31.26.194:44839)
0927 18:49:55.811438 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:55.811454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:55.811553 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:55.811894 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:55.811899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:55.811923 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:55.812306 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.812317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.813478 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.813481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.814979 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:55.814994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.816303 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.816308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:55.816556 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.816589 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:49:55.816694 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:49:55.816829 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":48,"thread_start_us":413,"threads_started":1}
I20260927 18:49:55.818274 19690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb9c5cfe2841d89db7e3f2aade9aae" candidate_uuid: "66034359f4e14449810e6d4f55d7b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b15e31407004141abfdcd826eed33e2" is_pre_election: true
I20260927 18:49:55.818725 19429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb9c5cfe2841d89db7e3f2aade9aae" candidate_uuid: "66034359f4e14449810e6d4f55d7b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c804978c3bc42a4ba9521605a21bab9" is_pre_election: true
I20260927 18:49:55.818847 19690 raft_consensus.cc:2454] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66034359f4e14449810e6d4f55d7b780 in term 0.
I20260927 18:49:55.819233 19429 raft_consensus.cc:2454] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66034359f4e14449810e6d4f55d7b780 in term 0.
I20260927 18:49:55.819620 19496 leader_election.cc:302] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b15e31407004141abfdcd826eed33e2, 66034359f4e14449810e6d4f55d7b780; no voters:
I20260927 18:49:55.820056 19760 raft_consensus.cc:2781] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:55.820206 19760 raft_consensus.cc:482] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:55.820330 19760 raft_consensus.cc:3037] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:55.822655 19760 raft_consensus.cc:504] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.823258 19760 leader_election.cc:288] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [CANDIDATE]: Term 1 election: Requested vote from peers 0b15e31407004141abfdcd826eed33e2 (127.31.26.195:37033), 4c804978c3bc42a4ba9521605a21bab9 (127.31.26.193:33349)
I20260927 18:49:55.823920 19690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb9c5cfe2841d89db7e3f2aade9aae" candidate_uuid: "66034359f4e14449810e6d4f55d7b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b15e31407004141abfdcd826eed33e2"
I20260927 18:49:55.823997 19429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb9c5cfe2841d89db7e3f2aade9aae" candidate_uuid: "66034359f4e14449810e6d4f55d7b780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c804978c3bc42a4ba9521605a21bab9"
I20260927 18:49:55.824152 19690 raft_consensus.cc:3037] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:55.824277 19429 raft_consensus.cc:3037] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:55.826715 19690 raft_consensus.cc:2454] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66034359f4e14449810e6d4f55d7b780 in term 1.
I20260927 18:49:55.826822 19429 raft_consensus.cc:2454] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66034359f4e14449810e6d4f55d7b780 in term 1.
I20260927 18:49:55.827261 19496 leader_election.cc:302] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b15e31407004141abfdcd826eed33e2, 66034359f4e14449810e6d4f55d7b780; no voters:
I20260927 18:49:55.827558 19760 raft_consensus.cc:2781] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:55.828017 19760 raft_consensus.cc:686] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 1 LEADER]: Becoming Leader. State: Replica: 66034359f4e14449810e6d4f55d7b780, State: Running, Role: LEADER
I20260927 18:49:55.828524 19760 consensus_queue.cc:237] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } }
I20260927 18:49:55.834503 19287 catalog_manager.cc:5495] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66034359f4e14449810e6d4f55d7b780 (127.31.26.194). New cstate: current_term: 1 leader_uuid: "66034359f4e14449810e6d4f55d7b780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66034359f4e14449810e6d4f55d7b780" member_type: VOTER last_known_addr { host: "127.31.26.194" port: 42173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 } health_report { overall_health: UNKNOWN } } }
I20260927 18:49:55.876789 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.871297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.31.26.194:42173)
0927 18:49:55.871505 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.871511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.871531 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:49:55.871682 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.871686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.871783 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.871949 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:55.872325 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.873025 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.874465 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.874911 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.875052 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.875584 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:55.876160 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:55.876333 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.876438 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.876520 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:49:55.885644 19429 raft_consensus.cc:1261] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 1 FOLLOWER]: Refusing update from remote peer 66034359f4e14449810e6d4f55d7b780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:55.885645 19690 raft_consensus.cc:1261] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 1 FOLLOWER]: Refusing update from remote peer 66034359f4e14449810e6d4f55d7b780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:55.886804 19764 consensus_queue.cc:1035] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b15e31407004141abfdcd826eed33e2" member_type: VOTER last_known_addr { host: "127.31.26.195" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:55.887233 19760 consensus_queue.cc:1035] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c804978c3bc42a4ba9521605a21bab9" member_type: VOTER last_known_addr { host: "127.31.26.193" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:49:55.894785 19736 tablet.cc:2369] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:55.916163 19288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:55.916479 19288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:55.920374 19288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 942dd7eadb5a4e7695b5a70d4f8e52f1: Sending DeleteTablet for 3 replicas of tablet 79eb9c5cfe2841d89db7e3f2aade9aae
I20260927 18:49:55.921659 19540 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eb9c5cfe2841d89db7e3f2aade9aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:55 UTC) from {username='slave'} at 127.0.0.1:55888
I20260927 18:49:55.921689 19670 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eb9c5cfe2841d89db7e3f2aade9aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:55 UTC) from {username='slave'} at 127.0.0.1:37182
I20260927 18:49:55.921693 19409 tablet_service.cc:1514] Processing DeleteTablet for tablet 79eb9c5cfe2841d89db7e3f2aade9aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:55 UTC) from {username='slave'} at 127.0.0.1:40110
I20260927 18:49:55.924091 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19347
I20260927 18:49:55.940896 19779 tablet_replica.cc:330] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: stopping tablet replica
I20260927 18:49:55.941210 19777 tablet_replica.cc:330] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: stopping tablet replica
I20260927 18:49:55.941700 19779 raft_consensus.cc:2229] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:55.942492 19779 raft_consensus.cc:2258] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:55.941821 19777 raft_consensus.cc:2229] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:55.943238 19778 tablet_replica.cc:330] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: stopping tablet replica
I20260927 18:49:55.943529 19777 raft_consensus.cc:2258] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:55.943684 19778 raft_consensus.cc:2229] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:55.944773 19779 ts_tablet_manager.cc:1905] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:55.946012 19777 ts_tablet_manager.cc:1905] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:55.948617 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:55.925721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.31.26.193:33349)
0927 18:49:55.925916 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:55.925923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:55.930990 (+ 5067us) server_negotiation.cc:407] Connection header received
0927 18:49:55.932044 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:55.932092 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:55.932221 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:55.932322 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:55.933872 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.935527 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.944151 (+ 8624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:55.944840 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:55.944965 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:55.946841 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:55.946853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:55.946872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:55.947651 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:55.947717 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:55.947750 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:55.947851 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:55.948113 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 18:49:55.948244 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:49:55.949726 19778 raft_consensus.cc:2258] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:55.950076 19469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:55.952267 19777 ts_tablet_manager.cc:1918] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:55.952455 19777 log.cc:1196] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-2/wal/wals/79eb9c5cfe2841d89db7e3f2aade9aae
I20260927 18:49:55.953022 19777 ts_tablet_manager.cc:1939] T 79eb9c5cfe2841d89db7e3f2aade9aae P 0b15e31407004141abfdcd826eed33e2: Deleting consensus metadata
I20260927 18:49:55.955066 19275 catalog_manager.cc:4841] TS 0b15e31407004141abfdcd826eed33e2 (127.31.26.195:37033): tablet 79eb9c5cfe2841d89db7e3f2aade9aae (table test-table [id=c5b40122bf6b415cad9fc4dbe3724f5e]) successfully deleted
I20260927 18:49:55.956666 19779 ts_tablet_manager.cc:1918] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:55.956950 19779 log.cc:1196] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Deleting WAL directory at /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534668899567-31851-0/minicluster-data/ts-1/wal/wals/79eb9c5cfe2841d89db7e3f2aade9aae
I20260927 18:49:55.957701 19779 ts_tablet_manager.cc:1939] T 79eb9c5cfe2841d89db7e3f2aade9aae P 66034359f4e14449810e6d4f55d7b780: Deleting consensus metadata
I20260927 18:49:55.959769 19273 catalog_manager.cc:4841] TS 66034359f4e14449810e6d4f55d7b780 (127.31.26.194:42173): tablet 79eb9c5cfe2841d89db7e3f2aade9aae (table test-table [id=c5b40122bf6b415cad9fc4dbe3724f5e]) successfully deleted
I20260927 18:49:56.047312 19778 ts_tablet_manager.cc:1905] T 79eb9c5cfe2841d89db7e3f2aade9aae P 4c804978c3bc42a4ba9521605a21bab9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:56.048780 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19347
W20260927 18:49:56.070297 19275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.26.193:33349 (error 108)
I20260927 18:49:56.071015 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19478
I20260927 18:49:56.072477 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.071919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40126)
0927 18:49:56.072166 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.072314 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 18:49:56.072788 19275 catalog_manager.cc:4584] TS 4c804978c3bc42a4ba9521605a21bab9 (127.31.26.193:33349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79eb9c5cfe2841d89db7e3f2aade9aae: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
I20260927 18:49:56.079923 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.072217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.31.26.194:42173)
0927 18:49:56.072423 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:56.072430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:56.072452 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:49:56.072519 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:56.072525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:56.072629 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:56.072754 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:56.073273 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:56.074527 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:56.076071 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:56.076810 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:56.076945 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:56.078149 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:56.078162 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:56.078183 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:56.079039 (+ 856us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:56.079112 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:56.079147 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:56.079263 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:56.079407 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:49:56.079529 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:49:56.080884 19600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:56.126539 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.126014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40142)
0927 18:49:56.126272 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.126366 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 18:49:56.127697 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.127315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40158)
0927 18:49:56.127474 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.127572 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:49:56.188650 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.188120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40170)
0927 18:49:56.188343 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.188439 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 18:49:56.189901 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.189508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40176)
0927 18:49:56.189683 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.189762 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:49:56.202028 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19478
I20260927 18:49:56.220929 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19608
I20260927 18:49:56.228408 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.221747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.31.26.195:37033)
0927 18:49:56.221979 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:56.221985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:56.222127 (+ 142us) server_negotiation.cc:407] Connection header received
0927 18:49:56.222372 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:56.222378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:56.222471 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:56.222593 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:56.222998 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:56.223758 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:56.225015 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:56.225654 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:56.225808 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:56.227000 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:56.227010 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:56.227025 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:49:56.227626 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:56.227684 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:49:56.227705 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:56.227839 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:56.227971 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 18:49:56.228085 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:49:56.229063 19730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:56.270746 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.270135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40186)
0927 18:49:56.270422 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.270555 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 18:49:56.271953 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.271469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40196)
0927 18:49:56.271706 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.271801 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 18:49:56.321774 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19608
I20260927 18:49:56.338969 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19257
I20260927 18:49:56.340167 19318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:56.450294 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:56.449639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.193:33349 (local address 127.0.0.1:40204)
0927 18:49:56.449989 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:56.450074 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.193:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260927 18:49:56.450306 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19257
2026-09-27T18:49:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2888 ms)
[----------] 6 tests from EncryptionPolicyTest (22541 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:56 dist-test-slave-rfd5 krb5kdc[19788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:56 dist-test-slave-rfd5 krb5kdc[19788](info): set up 2 sockets
Sep 27 18:49:56 dist-test-slave-rfd5 krb5kdc[19788](info): commencing operation
krb5kdc: starting...
W20260927 18:49:57.581758 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:49:57 dist-test-slave-rfd5 krb5kdc[19788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:49:57.852836 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:58.207329 19801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:58.207705 19801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:58.207870 19801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:58.215490 19801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:58.215684 19801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:58.215794 19801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:58.215903 19801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:58.215987 19801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:58.226414 19801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:49:58.227955 19801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:58.229440 19801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:58.242046 19806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:58.242063 19809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:58.242075 19807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:58.242908 19801 server_base.cc:1034] running on GCE node
I20260927 18:49:58.243781 19801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:58.245409 19801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:58.246678 19801 hybrid_clock.cc:648] HybridClock initialized: now 1790534998246615 us; error 43 us; skew 500 ppm
I20260927 18:49:58.247080 19801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:49:58 dist-test-slave-rfd5 krb5kdc[19788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:58.250556 19801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:49:58.252308 19801 webserver.cc:460] Webserver started at http://127.0.0.1:37373/ using document root <none> and password file <none>
I20260927 18:49:58.253278 19801 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:58.253449 19801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:58.253783 19801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:58.257025 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "55090d245fbf4e24b815e2b7c53fda3e"
format_stamp: "Formatted at 2026-09-27 18:49:58 on dist-test-slave-rfd5"
server_key: "61b0ebd7cd2917c63a2bd56551ce94cc"
server_key_iv: "44683970bde4b99f7807c5db71791367"
server_key_version: "encryptionkey@0"
I20260927 18:49:58.257997 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "55090d245fbf4e24b815e2b7c53fda3e"
format_stamp: "Formatted at 2026-09-27 18:49:58 on dist-test-slave-rfd5"
server_key: "61b0ebd7cd2917c63a2bd56551ce94cc"
server_key_iv: "44683970bde4b99f7807c5db71791367"
server_key_version: "encryptionkey@0"
I20260927 18:49:58.264559 19801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:58.268628 19816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:58.270414 19801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:58.270673 19801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "55090d245fbf4e24b815e2b7c53fda3e"
format_stamp: "Formatted at 2026-09-27 18:49:58 on dist-test-slave-rfd5"
server_key: "61b0ebd7cd2917c63a2bd56551ce94cc"
server_key_iv: "44683970bde4b99f7807c5db71791367"
server_key_version: "encryptionkey@0"
I20260927 18:49:58.270911 19801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:58.304323 19801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:58.309051 19801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:58.309520 19801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:58.329694 19801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39201
I20260927 18:49:58.329712 19867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39201 every 8 connection(s)
I20260927 18:49:58.331197 19801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:58.335402 19868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:58.339841 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19801
I20260927 18:49:58.340061 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:49:58.340775 31851 external_mini_cluster.cc:1428] Setting key 4b9ac1fde7033dec1001ff4f7be4bee6
I20260927 18:49:58.347450 19868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e: Bootstrap starting.
I20260927 18:49:58.350965 19868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:58.352109 19868 log.cc:824] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:58.355363 19868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e: No bootstrap required, opened a new log
Sep 27 18:49:58 dist-test-slave-rfd5 krb5kdc[19788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:49:58.363979 19868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55090d245fbf4e24b815e2b7c53fda3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260927 18:49:58.364579 19868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:58.364709 19868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55090d245fbf4e24b815e2b7c53fda3e, State: Initialized, Role: FOLLOWER
I20260927 18:49:58.365458 19868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55090d245fbf4e24b815e2b7c53fda3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260927 18:49:58.365775 19868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:58.365770 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:58.343009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53627 (local address 127.0.0.1:39201)
0927 18:49:58.343935 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:58.343949 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:58.344004 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:49:58.345979 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:58.346009 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:58.346480 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:58.346924 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:58.348571 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:58.350056 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:58.351497 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:58.352263 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:58.352490 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:58.356950 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:58.356959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:58.356978 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:58.357024 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:58.360517 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:58.361563 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:58.361572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:58.361580 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:58.361782 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:58.362340 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:58.362347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:58.362349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:58.363000 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:58.363381 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:58.363896 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 18:49:58.364273 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":430,"threads_started":1}
I20260927 18:49:58.365963 19868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:58.366179 19868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:58.369355 19868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55090d245fbf4e24b815e2b7c53fda3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260927 18:49:58.370015 19868 leader_election.cc:302] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55090d245fbf4e24b815e2b7c53fda3e; no voters:
I20260927 18:49:58.370510 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:49:58.370723 19868 leader_election.cc:288] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:58.370843 19873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:58.372951 19868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:58.382248 19873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [term 1 LEADER]: Becoming Leader. State: Replica: 55090d245fbf4e24b815e2b7c53fda3e, State: Running, Role: LEADER
I20260927 18:49:58.383400 19873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55090d245fbf4e24b815e2b7c53fda3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } }
I20260927 18:49:58.390470 19874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55090d245fbf4e24b815e2b7c53fda3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55090d245fbf4e24b815e2b7c53fda3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260927 18:49:58.390952 19874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:58.391489 19875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55090d245fbf4e24b815e2b7c53fda3e. Latest consensus state: current_term: 1 leader_uuid: "55090d245fbf4e24b815e2b7c53fda3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55090d245fbf4e24b815e2b7c53fda3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39201 } } }
I20260927 18:49:58.391757 19875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:58.394224 19890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:58.400262 19890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:58.405936 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:58.385046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40033 (local address 127.0.0.1:39201)
0927 18:49:58.385321 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:58.385331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:58.385875 (+ 544us) server_negotiation.cc:407] Connection header received
0927 18:49:58.386153 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:58.386161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:58.386278 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:58.386418 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:58.388932 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:58.390339 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:58.392348 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:58.393323 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:58.393477 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:58.397107 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:58.397118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:58.397128 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:58.397172 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:58.401510 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:58.402834 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:58.402841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:58.402845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:58.403011 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:58.403689 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:58.403695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:58.403699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:58.404171 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:58.404465 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:58.404992 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 18:49:58.405249 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:49:58.411131 19890 catalog_manager.cc:1347] Generated new cluster ID: 8c79e82b092e47a7bbaae7955ca7f5a1
I20260927 18:49:58.411357 19890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:58.428640 19890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:58.430795 19890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:58.448400 19890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55090d245fbf4e24b815e2b7c53fda3e: Generated new TSK 0
I20260927 18:49:58.449658 19890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:49:58.457778 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19801
I20260927 18:49:58.459245 19863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:49:58.554869 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19801
W20260927 18:49:58.574254 19786 connection.cc:535] client connection to 127.0.0.1:39201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39201): Connection reset by peer (error 104)
2026-09-27T18:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2096 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:58 dist-test-slave-rfd5 krb5kdc[19900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:58 dist-test-slave-rfd5 krb5kdc[19900](info): set up 2 sockets
Sep 27 18:49:58 dist-test-slave-rfd5 krb5kdc[19900](info): commencing operation
krb5kdc: starting...
W20260927 18:49:59.669618 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:49:59 dist-test-slave-rfd5 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:49:59.944455 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:00.296839 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:00.297233 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:00.297380 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:00.305676 19913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:00.305811 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:00.305902 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:00.306013 19913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:00.306105 19913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:00.316627 19913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:00.318073 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:00.319578 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:00.332289 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:00.332331 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:00.332389 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:00.334744 19913 server_base.cc:1034] running on GCE node
I20260927 18:50:00.335772 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:00.337664 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:00.338945 19913 hybrid_clock.cc:648] HybridClock initialized: now 1790535000338841 us; error 69 us; skew 500 ppm
I20260927 18:50:00.339478 19913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:00 dist-test-slave-rfd5 krb5kdc[19900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:00.343086 19913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:00.344909 19913 webserver.cc:460] Webserver started at http://127.0.0.1:41751/ using document root <none> and password file <none>
I20260927 18:50:00.345747 19913 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:00.345902 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:00.346237 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:00.349501 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "3bddba0a063f4b94b8037471c884e9bd"
format_stamp: "Formatted at 2026-09-27 18:50:00 on dist-test-slave-rfd5"
server_key: "b0ea45d9200a677191721e693bc95429"
server_key_iv: "9ff56811034ca9e5e2a6c4f2d3c41e55"
server_key_version: "encryptionkey@0"
I20260927 18:50:00.350355 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "3bddba0a063f4b94b8037471c884e9bd"
format_stamp: "Formatted at 2026-09-27 18:50:00 on dist-test-slave-rfd5"
server_key: "b0ea45d9200a677191721e693bc95429"
server_key_iv: "9ff56811034ca9e5e2a6c4f2d3c41e55"
server_key_version: "encryptionkey@0"
I20260927 18:50:00.355963 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:00.359807 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:00.361608 19913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:50:00.361819 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "3bddba0a063f4b94b8037471c884e9bd"
format_stamp: "Formatted at 2026-09-27 18:50:00 on dist-test-slave-rfd5"
server_key: "b0ea45d9200a677191721e693bc95429"
server_key_iv: "9ff56811034ca9e5e2a6c4f2d3c41e55"
server_key_version: "encryptionkey@0"
I20260927 18:50:00.362043 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:00.375881 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:00.380733 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:00.381037 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:00.400954 19913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39853
I20260927 18:50:00.400978 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39853 every 8 connection(s)
I20260927 18:50:00.402611 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:00.406725 19980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:00.410825 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 19913
I20260927 18:50:00.411020 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:00.411581 31851 external_mini_cluster.cc:1428] Setting key 9ac06ff30a204d5bbb58344311e37e03
I20260927 18:50:00.422019 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd: Bootstrap starting.
I20260927 18:50:00.426419 19980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:00.427788 19980 log.cc:824] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:00.428030 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:00.413809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48671 (local address 127.0.0.1:39853)
0927 18:50:00.414831 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:00.414848 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:00.415021 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:50:00.416986 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:00.417017 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:00.417649 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:00.418058 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:00.418937 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:00.420506 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:00.422617 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:00.423413 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:00.423577 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:00.425301 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:00.425310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:00.425327 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:00.425955 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:00.426073 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:00.426103 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:00.426249 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:00.426694 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 18:50:00.426877 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":505,"threads_started":1}
I20260927 18:50:00.431345 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd: No bootstrap required, opened a new log
I20260927 18:50:00.432705 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:00.440181 19980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bddba0a063f4b94b8037471c884e9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39853 } }
I20260927 18:50:00.440694 19980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:00.440832 19980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bddba0a063f4b94b8037471c884e9bd, State: Initialized, Role: FOLLOWER
I20260927 18:50:00.441705 19980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bddba0a063f4b94b8037471c884e9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39853 } }
I20260927 18:50:00.442021 19980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:00.442186 19980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:00.442396 19980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:00.445570 19980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bddba0a063f4b94b8037471c884e9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39853 } }
I20260927 18:50:00.446146 19980 leader_election.cc:302] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bddba0a063f4b94b8037471c884e9bd; no voters:
I20260927 18:50:00.447103 19980 leader_election.cc:288] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:00.447269 19990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:00.448789 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:00.437688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57291 (local address 127.0.0.1:39853)
0927 18:50:00.437910 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:00.437917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:00.441396 (+ 3479us) server_negotiation.cc:407] Connection header received
0927 18:50:00.441605 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:00.441609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:00.441687 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:00.441782 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:00.442478 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:00.443616 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:00.445317 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:00.445813 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:00.445935 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:00.447342 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:00.447351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:00.447358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:00.447384 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:00.447443 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:00.447470 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:00.447591 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:00.448011 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 18:50:00.448093 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":70}
I20260927 18:50:00.448782 19990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [term 1 LEADER]: Becoming Leader. State: Replica: 3bddba0a063f4b94b8037471c884e9bd, State: Running, Role: LEADER
I20260927 18:50:00.449843 19990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bddba0a063f4b94b8037471c884e9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39853 } }
I20260927 18:50:00.450057 19980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:00.453260 19992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bddba0a063f4b94b8037471c884e9bd. Latest consensus state: current_term: 1 leader_uuid: "3bddba0a063f4b94b8037471c884e9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bddba0a063f4b94b8037471c884e9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39853 } } }
I20260927 18:50:00.453305 19991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bddba0a063f4b94b8037471c884e9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bddba0a063f4b94b8037471c884e9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39853 } } }
I20260927 18:50:00.453663 19992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:00.453722 19991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:00.458222 19996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:00.465157 19996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:00.474999 19996 catalog_manager.cc:1347] Generated new cluster ID: df4ada98b0774444a1e87f4aa4132d3f
I20260927 18:50:00.475143 19996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:00.494858 19996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:00.496469 19996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:00.502737 19996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bddba0a063f4b94b8037471c884e9bd: Generated new TSK 0
I20260927 18:50:00.503597 19996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:00.513816 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19913
W20260927 18:50:00.515072 19975 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48671
W20260927 18:50:00.515966 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 19913: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:00.516163 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 19913
W20260927 18:50:00.530437 19898 connection.cc:535] client connection to 127.0.0.1:39853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39853): Connection reset by peer (error 104)
2026-09-27T18:50:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:00 dist-test-slave-rfd5 krb5kdc[20011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:00 dist-test-slave-rfd5 krb5kdc[20011](info): set up 2 sockets
Sep 27 18:50:00 dist-test-slave-rfd5 krb5kdc[20011](info): commencing operation
krb5kdc: starting...
W20260927 18:50:01.617646 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:01 dist-test-slave-rfd5 krb5kdc[20011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:01.885511 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:02.234154 20024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:02.234489 20024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:02.234681 20024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:02.242698 20024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:02.242859 20024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:02.242909 20024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:02.243197 20024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:02.243288 20024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:02.254031 20024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:02.255517 20024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:02.257294 20024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:02.271816 20030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:02.271883 20032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:02.271852 20029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:02.272826 20024 server_base.cc:1034] running on GCE node
I20260927 18:50:02.273838 20024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:02.275491 20024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:02.276736 20024 hybrid_clock.cc:648] HybridClock initialized: now 1790535002276693 us; error 39 us; skew 500 ppm
I20260927 18:50:02.277220 20024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:02 dist-test-slave-rfd5 krb5kdc[20011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:02.290005 20024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:02.292172 20024 webserver.cc:460] Webserver started at http://127.0.0.1:36069/ using document root <none> and password file <none>
I20260927 18:50:02.292997 20024 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:02.293249 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:02.293584 20024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:02.297019 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "a6ae973e114746fd8e75bc654363df7d"
format_stamp: "Formatted at 2026-09-27 18:50:02 on dist-test-slave-rfd5"
server_key: "5b96960a9e544f3a802435220a9cda04"
server_key_iv: "62fe9ed80c51c6d8cdb7f098c7374f63"
server_key_version: "encryptionkey@0"
I20260927 18:50:02.297999 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "a6ae973e114746fd8e75bc654363df7d"
format_stamp: "Formatted at 2026-09-27 18:50:02 on dist-test-slave-rfd5"
server_key: "5b96960a9e544f3a802435220a9cda04"
server_key_iv: "62fe9ed80c51c6d8cdb7f098c7374f63"
server_key_version: "encryptionkey@0"
I20260927 18:50:02.304701 20024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:50:02.308816 20039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:02.310551 20024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:50:02.310771 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "a6ae973e114746fd8e75bc654363df7d"
format_stamp: "Formatted at 2026-09-27 18:50:02 on dist-test-slave-rfd5"
server_key: "5b96960a9e544f3a802435220a9cda04"
server_key_iv: "62fe9ed80c51c6d8cdb7f098c7374f63"
server_key_version: "encryptionkey@0"
I20260927 18:50:02.310984 20024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:02.317405 20024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:02.322340 20024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:02.322676 20024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:02.343223 20024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36823
I20260927 18:50:02.343262 20090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36823 every 8 connection(s)
I20260927 18:50:02.344774 20024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:02.349035 20091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:02.352509 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20024
I20260927 18:50:02.352743 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:02.353339 31851 external_mini_cluster.cc:1428] Setting key 71bcbc20b47e6510aa0e1f0820b6f02e
I20260927 18:50:02.361955 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:02.355396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48675 (local address 127.0.0.1:36823)
0927 18:50:02.356267 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:02.356281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:02.356325 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:50:02.358186 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:02.358214 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:02.358637 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:02.359026 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:02.360212 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:02.360218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:02.360232 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:02.360248 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:02.360313 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:02.360332 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:02.360507 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:02.360637 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:50:02.360869 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":402,"threads_started":1}
I20260927 18:50:02.362450 20091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d: Bootstrap starting.
I20260927 18:50:02.365945 20091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:02.366214 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:02.367327 20091 log.cc:824] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:02.371049 20091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d: No bootstrap required, opened a new log
I20260927 18:50:02.374518 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:02.371651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54501 (local address 127.0.0.1:36823)
0927 18:50:02.371857 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:02.371867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:02.372391 (+ 524us) server_negotiation.cc:407] Connection header received
0927 18:50:02.372583 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:02.372586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:02.372637 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:02.372729 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:02.373754 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:02.373759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:02.373762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:02.373779 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:02.373820 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:02.373843 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:02.373949 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:02.374081 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 18:50:02.374136 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:50:02.379945 20091 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae973e114746fd8e75bc654363df7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260927 18:50:02.380415 20091 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:02.380543 20091 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ae973e114746fd8e75bc654363df7d, State: Initialized, Role: FOLLOWER
I20260927 18:50:02.381258 20091 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae973e114746fd8e75bc654363df7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260927 18:50:02.381620 20091 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:02.381796 20091 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:02.382002 20091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:02.384267 20091 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae973e114746fd8e75bc654363df7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260927 18:50:02.384832 20091 leader_election.cc:302] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ae973e114746fd8e75bc654363df7d; no voters:
I20260927 18:50:02.385553 20091 leader_election.cc:288] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:02.385773 20101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:02.386520 20101 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [term 1 LEADER]: Becoming Leader. State: Replica: a6ae973e114746fd8e75bc654363df7d, State: Running, Role: LEADER
I20260927 18:50:02.387130 20101 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae973e114746fd8e75bc654363df7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } }
I20260927 18:50:02.388010 20091 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:02.390944 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ae973e114746fd8e75bc654363df7d. Latest consensus state: current_term: 1 leader_uuid: "a6ae973e114746fd8e75bc654363df7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae973e114746fd8e75bc654363df7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } } }
I20260927 18:50:02.391613 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:02.390934 20102 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ae973e114746fd8e75bc654363df7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae973e114746fd8e75bc654363df7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36823 } } }
I20260927 18:50:02.392256 20102 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:02.394250 20110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:02.399441 20110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:02.409690 20110 catalog_manager.cc:1347] Generated new cluster ID: 44813f180b3541cabfbe3eb203babb2f
I20260927 18:50:02.409837 20110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:02.422752 20110 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:02.423305 20110 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:02.432152 20110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ae973e114746fd8e75bc654363df7d: Generated new TSK 0
I20260927 18:50:02.433207 20110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:02.451300 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20024
W20260927 18:50:02.452996 20086 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48675
W20260927 18:50:02.453887 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20024: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:02.454039 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20024
2026-09-27T18:50:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:02 dist-test-slave-rfd5 krb5kdc[20122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:02 dist-test-slave-rfd5 krb5kdc[20122](info): set up 2 sockets
Sep 27 18:50:02 dist-test-slave-rfd5 krb5kdc[20122](info): commencing operation
krb5kdc: starting...
W20260927 18:50:03.573578 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:03 dist-test-slave-rfd5 krb5kdc[20122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:03.835791 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:04.180704 20135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:04.181085 20135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:04.181281 20135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:04.188802 20135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:04.188992 20135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:04.189121 20135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:04.189253 20135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:04.189334 20135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:04.199854 20135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:04.201432 20135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:04.202847 20135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:04.215065 20140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:04.215129 20141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:04.215059 20143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:04.216028 20135 server_base.cc:1034] running on GCE node
I20260927 18:50:04.216878 20135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:04.218477 20135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:04.219696 20135 hybrid_clock.cc:648] HybridClock initialized: now 1790535004219657 us; error 33 us; skew 500 ppm
I20260927 18:50:04.220062 20135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:04 dist-test-slave-rfd5 krb5kdc[20122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:04.223335 20135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:04.224865 20135 webserver.cc:460] Webserver started at http://127.0.0.1:46261/ using document root <none> and password file <none>
I20260927 18:50:04.225646 20135 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:04.225802 20135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:04.226198 20135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:04.229406 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "b1b95839d1ae4bb88153bcd8c45b637c"
format_stamp: "Formatted at 2026-09-27 18:50:04 on dist-test-slave-rfd5"
server_key: "b2d3ac672226d539fae81ff7b209f169"
server_key_iv: "a4db6b68f758242da09a2ccffaf56d48"
server_key_version: "encryptionkey@0"
I20260927 18:50:04.230242 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "b1b95839d1ae4bb88153bcd8c45b637c"
format_stamp: "Formatted at 2026-09-27 18:50:04 on dist-test-slave-rfd5"
server_key: "b2d3ac672226d539fae81ff7b209f169"
server_key_iv: "a4db6b68f758242da09a2ccffaf56d48"
server_key_version: "encryptionkey@0"
I20260927 18:50:04.235874 20135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:50:04.239632 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:04.241076 20135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:50:04.241389 20135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "b1b95839d1ae4bb88153bcd8c45b637c"
format_stamp: "Formatted at 2026-09-27 18:50:04 on dist-test-slave-rfd5"
server_key: "b2d3ac672226d539fae81ff7b209f169"
server_key_iv: "a4db6b68f758242da09a2ccffaf56d48"
server_key_version: "encryptionkey@0"
I20260927 18:50:04.241639 20135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:04.264807 20135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:04.269699 20135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:04.270043 20135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:04.291033 20135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42857
I20260927 18:50:04.291051 20201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42857 every 8 connection(s)
I20260927 18:50:04.292573 20135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:04.296703 20202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:04.302276 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20135
I20260927 18:50:04.302475 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:04.302913 31851 external_mini_cluster.cc:1428] Setting key 98f9864d080cff13d0c235dd9823db43
I20260927 18:50:04.309507 20202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c: Bootstrap starting.
I20260927 18:50:04.313663 20202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:04.314889 20202 log.cc:824] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:04.318226 20202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c: No bootstrap required, opened a new log
Sep 27 18:50:04 dist-test-slave-rfd5 krb5kdc[20122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:50:04.326792 20202 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260927 18:50:04.327230 20202 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:04.327374 20202 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b95839d1ae4bb88153bcd8c45b637c, State: Initialized, Role: FOLLOWER
I20260927 18:50:04.328096 20202 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260927 18:50:04.328394 20202 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:04.328567 20202 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:04.328771 20202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:04.329052 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:04.304936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49573 (local address 127.0.0.1:42857)
0927 18:50:04.305952 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:04.305970 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:04.306029 (+ 59us) server_negotiation.cc:407] Connection header received
0927 18:50:04.308518 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:04.308552 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:04.309195 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:04.309703 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:04.311172 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:04.313294 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:04.314726 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:04.315857 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:04.316092 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:04.320565 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:04.320574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:04.320592 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:04.320643 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:04.324155 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:04.325017 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:04.325025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:04.325032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:04.325311 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:04.325759 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:04.325770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:04.325773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:04.326449 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:04.326744 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:04.327256 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 18:50:04.327753 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":490,"threads_started":1}
I20260927 18:50:04.331887 20202 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260927 18:50:04.332432 20202 leader_election.cc:302] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b95839d1ae4bb88153bcd8c45b637c; no voters:
I20260927 18:50:04.333209 20202 leader_election.cc:288] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:04.333351 20207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:04.333639 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:04.335414 20202 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:04.342397 20207 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [term 1 LEADER]: Becoming Leader. State: Replica: b1b95839d1ae4bb88153bcd8c45b637c, State: Running, Role: LEADER
I20260927 18:50:04.343369 20207 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } }
I20260927 18:50:04.347425 20208 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } } }
I20260927 18:50:04.347890 20208 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:04.348562 20209 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b95839d1ae4bb88153bcd8c45b637c. Latest consensus state: current_term: 1 leader_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b95839d1ae4bb88153bcd8c45b637c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42857 } } }
I20260927 18:50:04.348858 20209 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:04.353863 20227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:04.360003 20227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:04.366272 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:04.344014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43943 (local address 127.0.0.1:42857)
0927 18:50:04.344255 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:04.344265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:04.345517 (+ 1252us) server_negotiation.cc:407] Connection header received
0927 18:50:04.345920 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:04.345927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:04.346006 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:04.346101 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:04.349487 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:04.351102 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:04.353888 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:04.354650 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:04.354835 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:04.358430 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:04.358438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:04.358447 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:04.358495 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:04.362881 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:04.363936 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:04.363940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:04.363943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:04.364084 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:04.364565 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:04.364569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:04.364571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:04.364928 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:04.365235 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:04.365620 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 18:50:04.365870 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:50:04.370293 20227 catalog_manager.cc:1347] Generated new cluster ID: 4798033ba6e04349a245c2b5573af365
I20260927 18:50:04.370433 20227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:04.385882 20227 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:04.387312 20227 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:04.395328 20227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b95839d1ae4bb88153bcd8c45b637c: Generated new TSK 0
I20260927 18:50:04.396317 20227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:04.405958 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20135
I20260927 18:50:04.407150 20197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:04.499858 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20135
2026-09-27T18:50:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:04 dist-test-slave-rfd5 krb5kdc[20234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:04 dist-test-slave-rfd5 krb5kdc[20234](info): set up 2 sockets
Sep 27 18:50:04 dist-test-slave-rfd5 krb5kdc[20234](info): commencing operation
krb5kdc: starting...
W20260927 18:50:05.613624 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:05 dist-test-slave-rfd5 krb5kdc[20234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:05.881827 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:06.244940 20247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:06.245371 20247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:06.245549 20247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:06.253707 20247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:06.253849 20247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:06.253937 20247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:06.254029 20247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:06.254069 20247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:06.264569 20247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:06.266073 20247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:06.267537 20247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:06.280295 20252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:06.280242 20255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:06.280211 20253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:06.281884 20247 server_base.cc:1034] running on GCE node
I20260927 18:50:06.282918 20247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:06.284547 20247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:06.285825 20247 hybrid_clock.cc:648] HybridClock initialized: now 1790535006285732 us; error 63 us; skew 500 ppm
I20260927 18:50:06.286355 20247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:06 dist-test-slave-rfd5 krb5kdc[20234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:06.290136 20247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:06.291893 20247 webserver.cc:460] Webserver started at http://127.0.0.1:39409/ using document root <none> and password file <none>
I20260927 18:50:06.292791 20247 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:06.292960 20247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:06.293413 20247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:06.298084 20247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4"
format_stamp: "Formatted at 2026-09-27 18:50:06 on dist-test-slave-rfd5"
server_key: "d47370a5140950f3959e58751ad354db"
server_key_iv: "1be65f667239cc99827d0c8a37f0a61a"
server_key_version: "encryptionkey@0"
I20260927 18:50:06.299239 20247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4"
format_stamp: "Formatted at 2026-09-27 18:50:06 on dist-test-slave-rfd5"
server_key: "d47370a5140950f3959e58751ad354db"
server_key_iv: "1be65f667239cc99827d0c8a37f0a61a"
server_key_version: "encryptionkey@0"
I20260927 18:50:06.305546 20247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 18:50:06.309823 20262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:06.311481 20247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:50:06.311741 20247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4"
format_stamp: "Formatted at 2026-09-27 18:50:06 on dist-test-slave-rfd5"
server_key: "d47370a5140950f3959e58751ad354db"
server_key_iv: "1be65f667239cc99827d0c8a37f0a61a"
server_key_version: "encryptionkey@0"
I20260927 18:50:06.311962 20247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:06.338023 20247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:06.343112 20247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:06.343466 20247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:06.365660 20247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42797
I20260927 18:50:06.365695 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42797 every 8 connection(s)
I20260927 18:50:06.367360 20247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:06.369486 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20247
I20260927 18:50:06.369720 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:06.370307 31851 external_mini_cluster.cc:1428] Setting key fe595a8f3e237ad9bfb4725f30f97ef1
I20260927 18:50:06.372383 20314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:06.384425 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:06.372426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48413 (local address 127.0.0.1:42797)
0927 18:50:06.373388 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:06.373401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:06.373454 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:50:06.375189 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:06.375214 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:06.375666 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:06.376118 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:06.376707 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:06.378180 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:06.379608 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:06.380329 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:06.380538 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:06.381871 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:06.381879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:06.381894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:06.382564 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:06.382655 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:06.382682 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:06.382837 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:06.383067 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 18:50:06.383382 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":415,"threads_started":1}
I20260927 18:50:06.384727 20314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4: Bootstrap starting.
I20260927 18:50:06.388531 20314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:06.388718 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:06.389787 20314 log.cc:824] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:06.392970 20314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4: No bootstrap required, opened a new log
I20260927 18:50:06.401309 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:06.393847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42339 (local address 127.0.0.1:42797)
0927 18:50:06.394052 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:06.394058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:06.394568 (+ 510us) server_negotiation.cc:407] Connection header received
0927 18:50:06.394770 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:06.394776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:06.394861 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:06.394997 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:06.395693 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:06.396859 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:06.398310 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:06.399070 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:06.399213 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:06.400379 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:06.400386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:06.400391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:06.400421 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:06.400472 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:06.400492 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:06.400595 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:06.400780 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 18:50:06.400899 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:50:06.403340 20314 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260927 18:50:06.403736 20314 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:06.403806 20314 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1d8d1928f74e01ac7563f54e4ac3b4, State: Initialized, Role: FOLLOWER
I20260927 18:50:06.404414 20314 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260927 18:50:06.404629 20314 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:06.404731 20314 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:06.404865 20314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:06.407570 20314 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260927 18:50:06.408087 20314 leader_election.cc:302] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1d8d1928f74e01ac7563f54e4ac3b4; no voters:
I20260927 18:50:06.408656 20314 leader_election.cc:288] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:06.408851 20324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:06.409663 20324 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [term 1 LEADER]: Becoming Leader. State: Replica: bd1d8d1928f74e01ac7563f54e4ac3b4, State: Running, Role: LEADER
I20260927 18:50:06.410272 20324 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260927 18:50:06.410966 20314 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:06.413726 20325 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260927 18:50:06.414175 20325 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:06.414512 20326 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1d8d1928f74e01ac7563f54e4ac3b4. Latest consensus state: current_term: 1 leader_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d8d1928f74e01ac7563f54e4ac3b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260927 18:50:06.415004 20326 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:06.427023 20330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:06.433465 20330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:06.444739 20330 catalog_manager.cc:1347] Generated new cluster ID: c641c9ea84da4909b64a3258f77b8a50
I20260927 18:50:06.444916 20330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:06.456173 20330 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:06.457779 20330 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:06.463565 20330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd1d8d1928f74e01ac7563f54e4ac3b4: Generated new TSK 0
I20260927 18:50:06.464519 20330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:06.481185 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20247
W20260927 18:50:06.482591 20309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48413
W20260927 18:50:06.483419 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:06.483588 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20247
2026-09-27T18:50:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:06 dist-test-slave-rfd5 krb5kdc[20345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:06 dist-test-slave-rfd5 krb5kdc[20345](info): set up 2 sockets
Sep 27 18:50:06 dist-test-slave-rfd5 krb5kdc[20345](info): commencing operation
krb5kdc: starting...
W20260927 18:50:07.615031 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:07 dist-test-slave-rfd5 krb5kdc[20345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:07.878841 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:08.225363 20358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:08.225719 20358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:08.225907 20358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:08.233510 20358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:08.233677 20358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:08.233793 20358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:08.233906 20358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:08.233986 20358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:08.244789 20358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:08.246258 20358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:08.247665 20358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:08.260334 20366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:08.260285 20363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:08.260391 20364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:08.261854 20358 server_base.cc:1034] running on GCE node
I20260927 18:50:08.262779 20358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:08.264344 20358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:08.265563 20358 hybrid_clock.cc:648] HybridClock initialized: now 1790535008265503 us; error 40 us; skew 500 ppm
I20260927 18:50:08.265944 20358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:08 dist-test-slave-rfd5 krb5kdc[20345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:08.278266 20358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:08.280136 20358 webserver.cc:460] Webserver started at http://127.0.0.1:38107/ using document root <none> and password file <none>
I20260927 18:50:08.280934 20358 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:08.281051 20358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:08.281414 20358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:08.284660 20358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "ddd64cc7a42e44649936811366c3b442"
format_stamp: "Formatted at 2026-09-27 18:50:08 on dist-test-slave-rfd5"
server_key: "018c34f4933a68a05a4b8ec45cca3dd7"
server_key_iv: "c7d7d780b31bfff89c41340400261d06"
server_key_version: "encryptionkey@0"
I20260927 18:50:08.285571 20358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "ddd64cc7a42e44649936811366c3b442"
format_stamp: "Formatted at 2026-09-27 18:50:08 on dist-test-slave-rfd5"
server_key: "018c34f4933a68a05a4b8ec45cca3dd7"
server_key_iv: "c7d7d780b31bfff89c41340400261d06"
server_key_version: "encryptionkey@0"
I20260927 18:50:08.291352 20358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:08.295441 20373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:08.297039 20358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:50:08.297326 20358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "ddd64cc7a42e44649936811366c3b442"
format_stamp: "Formatted at 2026-09-27 18:50:08 on dist-test-slave-rfd5"
server_key: "018c34f4933a68a05a4b8ec45cca3dd7"
server_key_iv: "c7d7d780b31bfff89c41340400261d06"
server_key_version: "encryptionkey@0"
I20260927 18:50:08.297559 20358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:08.304044 20358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:08.309887 20358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:08.310261 20358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:08.328951 20358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37751
I20260927 18:50:08.328987 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37751 every 8 connection(s)
I20260927 18:50:08.330677 20358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:08.334911 20425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:08.336831 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20358
I20260927 18:50:08.337014 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:08.337483 31851 external_mini_cluster.cc:1428] Setting key 2ba61edeb910428a7061a4ee76e017fd
I20260927 18:50:08.345789 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:08.339477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37651 (local address 127.0.0.1:37751)
0927 18:50:08.340360 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:08.340377 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:08.340424 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:50:08.342129 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:08.342155 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:08.342551 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:08.342962 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:08.344044 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:08.344050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:08.344065 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:08.344080 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:08.344138 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:08.344157 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:08.344405 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:08.344543 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:50:08.344748 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":392,"threads_started":1}
I20260927 18:50:08.349318 20425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442: Bootstrap starting.
I20260927 18:50:08.349915 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:08.352589 20425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:08.353790 20425 log.cc:824] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:08.357043 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:08.354406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33397 (local address 127.0.0.1:37751)
0927 18:50:08.354578 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:08.354582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:08.355075 (+ 493us) server_negotiation.cc:407] Connection header received
0927 18:50:08.355312 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:08.355318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:08.355374 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:08.355489 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:08.356422 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:08.356430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:08.356436 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:08.356454 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:08.356499 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:08.356519 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:08.356636 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:08.356715 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:50:08.356772 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:50:08.357368 20425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442: No bootstrap required, opened a new log
I20260927 18:50:08.365386 20425 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd64cc7a42e44649936811366c3b442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260927 18:50:08.365828 20425 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:08.365929 20425 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd64cc7a42e44649936811366c3b442, State: Initialized, Role: FOLLOWER
I20260927 18:50:08.366500 20425 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd64cc7a42e44649936811366c3b442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260927 18:50:08.366750 20425 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:08.366907 20425 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:08.367065 20425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:08.369577 20425 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd64cc7a42e44649936811366c3b442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260927 18:50:08.370096 20425 leader_election.cc:302] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddd64cc7a42e44649936811366c3b442; no voters:
I20260927 18:50:08.370735 20425 leader_election.cc:288] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:08.370878 20435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:08.371968 20435 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [term 1 LEADER]: Becoming Leader. State: Replica: ddd64cc7a42e44649936811366c3b442, State: Running, Role: LEADER
I20260927 18:50:08.372493 20435 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd64cc7a42e44649936811366c3b442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260927 18:50:08.372879 20425 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:08.375294 20437 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd64cc7a42e44649936811366c3b442. Latest consensus state: current_term: 1 leader_uuid: "ddd64cc7a42e44649936811366c3b442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd64cc7a42e44649936811366c3b442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } } }
I20260927 18:50:08.375742 20437 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:08.376304 20436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddd64cc7a42e44649936811366c3b442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd64cc7a42e44649936811366c3b442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } } }
I20260927 18:50:08.376608 20436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:08.379693 20444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:08.385865 20444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:08.395855 20444 catalog_manager.cc:1347] Generated new cluster ID: 225287613f6743588c8038351ab58c5f
I20260927 18:50:08.396013 20444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:08.405726 20444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:08.406265 20444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:08.417817 20444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddd64cc7a42e44649936811366c3b442: Generated new TSK 0
I20260927 18:50:08.418818 20444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:08.435602 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20358
W20260927 18:50:08.436853 20420 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37651
W20260927 18:50:08.437810 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:08.438010 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20358
2026-09-27T18:50:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:08 dist-test-slave-rfd5 krb5kdc[20456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:08 dist-test-slave-rfd5 krb5kdc[20456](info): set up 2 sockets
Sep 27 18:50:08 dist-test-slave-rfd5 krb5kdc[20456](info): commencing operation
krb5kdc: starting...
W20260927 18:50:09.541814 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:09 dist-test-slave-rfd5 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:09.817255 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:10.167677 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:10.167986 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:10.168179 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:10.175828 20469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:10.175963 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:10.176079 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:10.176187 20469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:10.176272 20469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:10.186584 20469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:10.188032 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:10.189491 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:10.202275 20474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:10.202314 20477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:10.202569 20475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:10.203855 20469 server_base.cc:1034] running on GCE node
I20260927 18:50:10.204886 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:10.206578 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:10.207839 20469 hybrid_clock.cc:648] HybridClock initialized: now 1790535010207793 us; error 43 us; skew 500 ppm
I20260927 18:50:10.208244 20469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:10 dist-test-slave-rfd5 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:10.211951 20469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:10.213719 20469 webserver.cc:460] Webserver started at http://127.0.0.1:45187/ using document root <none> and password file <none>
I20260927 18:50:10.214586 20469 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:10.214787 20469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:10.215138 20469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:10.218521 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "c1dac0377138484aaf81531e747d0e8a"
format_stamp: "Formatted at 2026-09-27 18:50:10 on dist-test-slave-rfd5"
server_key: "0c89f997779b613d8ecd3a92cbdf02bc"
server_key_iv: "9c0804fd534b678f1e09281b282f3720"
server_key_version: "encryptionkey@0"
I20260927 18:50:10.219456 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "c1dac0377138484aaf81531e747d0e8a"
format_stamp: "Formatted at 2026-09-27 18:50:10 on dist-test-slave-rfd5"
server_key: "0c89f997779b613d8ecd3a92cbdf02bc"
server_key_iv: "9c0804fd534b678f1e09281b282f3720"
server_key_version: "encryptionkey@0"
I20260927 18:50:10.225335 20469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:50:10.229063 20484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:10.230600 20469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:10.230840 20469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "c1dac0377138484aaf81531e747d0e8a"
format_stamp: "Formatted at 2026-09-27 18:50:10 on dist-test-slave-rfd5"
server_key: "0c89f997779b613d8ecd3a92cbdf02bc"
server_key_iv: "9c0804fd534b678f1e09281b282f3720"
server_key_version: "encryptionkey@0"
I20260927 18:50:10.231076 20469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:10.260376 20469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:10.265532 20469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:10.265867 20469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:10.285072 20469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40001
I20260927 18:50:10.285135 20535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40001 every 8 connection(s)
I20260927 18:50:10.286602 20469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:10.290769 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:10.293820 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20469
I20260927 18:50:10.294013 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:10.294461 31851 external_mini_cluster.cc:1428] Setting key 26a3d3bd5db14b17a4e710b8e1f52896
I20260927 18:50:10.302732 20536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a: Bootstrap starting.
I20260927 18:50:10.306039 20536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:10.307107 20536 log.cc:824] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:50:10 dist-test-slave-rfd5 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:50:10.310173 20536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a: No bootstrap required, opened a new log
I20260927 18:50:10.319226 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:10.296534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:39003 (local address 127.0.0.1:40001)
0927 18:50:10.297445 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:10.297458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:10.297501 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:50:10.299269 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:10.299293 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:10.299747 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:10.300180 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:10.301769 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:10.303147 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:10.304432 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:10.305176 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:10.305359 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:10.309653 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:10.309661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:10.309676 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:10.309714 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:10.313560 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:10.314748 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:10.314763 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:10.314772 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:10.315005 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:10.315597 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:10.315605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:10.315611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:10.316443 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:10.316791 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:10.317297 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 18:50:10.317704 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":339,"threads_started":1}
I20260927 18:50:10.321514 20536 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dac0377138484aaf81531e747d0e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40001 } }
I20260927 18:50:10.321928 20536 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:10.322045 20536 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1dac0377138484aaf81531e747d0e8a, State: Initialized, Role: FOLLOWER
I20260927 18:50:10.322659 20536 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dac0377138484aaf81531e747d0e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40001 } }
I20260927 18:50:10.322947 20536 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:10.323097 20536 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:10.323293 20536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:10.323884 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:10.325754 20536 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dac0377138484aaf81531e747d0e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40001 } }
I20260927 18:50:10.326292 20536 leader_election.cc:302] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1dac0377138484aaf81531e747d0e8a; no voters:
I20260927 18:50:10.327023 20536 leader_election.cc:288] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:10.327176 20543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:10.329813 20536 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:10.333442 20543 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [term 1 LEADER]: Becoming Leader. State: Replica: c1dac0377138484aaf81531e747d0e8a, State: Running, Role: LEADER
I20260927 18:50:10.334321 20543 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dac0377138484aaf81531e747d0e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40001 } }
I20260927 18:50:10.338075 20546 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1dac0377138484aaf81531e747d0e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dac0377138484aaf81531e747d0e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40001 } } }
I20260927 18:50:10.338591 20546 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:10.341058 20547 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1dac0377138484aaf81531e747d0e8a. Latest consensus state: current_term: 1 leader_uuid: "c1dac0377138484aaf81531e747d0e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dac0377138484aaf81531e747d0e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40001 } } }
I20260927 18:50:10.342283 20547 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:10.350082 20553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:10.357430 20553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:10.371704 20553 catalog_manager.cc:1347] Generated new cluster ID: c7386c70f37c45efb34d854f8300943e
I20260927 18:50:10.371829 20553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:10.375155 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:10.335331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:58585 (local address 127.0.0.1:40001)
0927 18:50:10.335658 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:10.335669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:10.348567 (+ 12898us) server_negotiation.cc:407] Connection header received
0927 18:50:10.348944 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:10.348955 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:10.349072 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:10.349186 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:10.354330 (+ 5144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:10.355616 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:10.357558 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:10.358080 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:10.358235 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:10.363228 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:10.363240 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:10.363246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:10.363295 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:10.369576 (+ 6281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:10.370763 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:10.370775 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:10.370779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:10.370944 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:10.373323 (+ 2379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:10.373329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:10.373333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:10.373771 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:10.374048 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:10.374565 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 18:50:10.374768 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":166}
I20260927 18:50:10.389006 20553 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:10.390434 20553 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:10.406596 20553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1dac0377138484aaf81531e747d0e8a: Generated new TSK 0
I20260927 18:50:10.407584 20553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:10.414832 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20469
I20260927 18:50:10.416074 20531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:10.510089 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20469
W20260927 18:50:10.527614 20454 connection.cc:535] client connection to 127.0.0.1:40001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40001): Connection reset by peer (error 104)
2026-09-27T18:50:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:10 dist-test-slave-rfd5 krb5kdc[20569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:10 dist-test-slave-rfd5 krb5kdc[20569](info): set up 2 sockets
Sep 27 18:50:10 dist-test-slave-rfd5 krb5kdc[20569](info): commencing operation
krb5kdc: starting...
W20260927 18:50:11.613718 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:11 dist-test-slave-rfd5 krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:11.880330 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:12.231454 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:12.231777 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:12.231869 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:12.239524 20582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:12.239656 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:12.239704 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:12.239765 20582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:12.239810 20582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:12.250324 20582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:12.251693 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:12.253046 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:12.266355 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:12.266536 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:12.266579 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:12.267508 20582 server_base.cc:1034] running on GCE node
I20260927 18:50:12.268383 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:12.270051 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:12.271286 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790535012271241 us; error 39 us; skew 500 ppm
I20260927 18:50:12.271641 20582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:12 dist-test-slave-rfd5 krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:12.275213 20582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:12.276932 20582 webserver.cc:460] Webserver started at http://127.0.0.1:36309/ using document root <none> and password file <none>
I20260927 18:50:12.277727 20582 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:12.277872 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:12.278218 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:12.281497 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "c688cae3b60e462e88df69383fe0ffe8"
format_stamp: "Formatted at 2026-09-27 18:50:12 on dist-test-slave-rfd5"
server_key: "995278bf34307b472030e5b1dea5c69c"
server_key_iv: "d420c19b898c5fb7952f19827a4b6330"
server_key_version: "encryptionkey@0"
I20260927 18:50:12.282347 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "c688cae3b60e462e88df69383fe0ffe8"
format_stamp: "Formatted at 2026-09-27 18:50:12 on dist-test-slave-rfd5"
server_key: "995278bf34307b472030e5b1dea5c69c"
server_key_iv: "d420c19b898c5fb7952f19827a4b6330"
server_key_version: "encryptionkey@0"
I20260927 18:50:12.288069 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:50:12.292029 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:12.293807 20582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:12.294031 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "c688cae3b60e462e88df69383fe0ffe8"
format_stamp: "Formatted at 2026-09-27 18:50:12 on dist-test-slave-rfd5"
server_key: "995278bf34307b472030e5b1dea5c69c"
server_key_iv: "d420c19b898c5fb7952f19827a4b6330"
server_key_version: "encryptionkey@0"
I20260927 18:50:12.294284 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:12.318238 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:12.322857 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:12.323191 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:12.344209 20582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39687
I20260927 18:50:12.344251 20648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39687 every 8 connection(s)
I20260927 18:50:12.345813 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:12.347370 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20582
I20260927 18:50:12.347564 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:12.348060 31851 external_mini_cluster.cc:1428] Setting key b37852951e1a516d0a1acf9bf48fecb6
I20260927 18:50:12.352223 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:12.363065 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:12.350069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:41039 (local address 127.0.0.1:39687)
0927 18:50:12.351082 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:12.351096 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:12.351144 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:50:12.353076 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:12.353175 (+ 99us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:12.353652 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:12.354031 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:12.354744 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:12.356122 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:12.358111 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:12.358858 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:12.359006 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:12.360423 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:12.360430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:12.360446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:12.361029 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:12.361295 (+ 266us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:12.361329 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:12.361455 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:12.361798 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 18:50:12.362015 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":492,"threads_started":1}
I20260927 18:50:12.365458 20649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8: Bootstrap starting.
I20260927 18:50:12.367300 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:12.368940 20649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:12.370052 20649 log.cc:824] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:12.373682 20649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8: No bootstrap required, opened a new log
I20260927 18:50:12.378718 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:12.371646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:48133 (local address 127.0.0.1:39687)
0927 18:50:12.371834 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:12.371840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:12.372386 (+ 546us) server_negotiation.cc:407] Connection header received
0927 18:50:12.372549 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:12.372554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:12.372640 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:12.372757 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:12.373339 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:12.374500 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:12.375839 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:12.376568 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:12.376730 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:12.377885 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:12.377891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:12.377896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:12.377920 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:12.377974 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:12.377999 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:12.378126 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:12.378309 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:50:12.378371 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:50:12.383009 20649 raft_consensus.cc:348] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c688cae3b60e462e88df69383fe0ffe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39687 } }
I20260927 18:50:12.383498 20649 raft_consensus.cc:374] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:12.383628 20649 raft_consensus.cc:729] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688cae3b60e462e88df69383fe0ffe8, State: Initialized, Role: FOLLOWER
I20260927 18:50:12.384316 20649 consensus_queue.cc:260] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c688cae3b60e462e88df69383fe0ffe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39687 } }
I20260927 18:50:12.384578 20649 raft_consensus.cc:388] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:12.384742 20649 raft_consensus.cc:482] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:12.384931 20649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:12.387282 20649 raft_consensus.cc:504] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c688cae3b60e462e88df69383fe0ffe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39687 } }
I20260927 18:50:12.387847 20649 leader_election.cc:302] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c688cae3b60e462e88df69383fe0ffe8; no voters:
I20260927 18:50:12.388513 20649 leader_election.cc:288] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:12.388684 20659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:12.389460 20659 raft_consensus.cc:686] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [term 1 LEADER]: Becoming Leader. State: Replica: c688cae3b60e462e88df69383fe0ffe8, State: Running, Role: LEADER
I20260927 18:50:12.390146 20659 consensus_queue.cc:237] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c688cae3b60e462e88df69383fe0ffe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39687 } }
I20260927 18:50:12.390793 20649 sys_catalog.cc:564] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:12.393326 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c688cae3b60e462e88df69383fe0ffe8. Latest consensus state: current_term: 1 leader_uuid: "c688cae3b60e462e88df69383fe0ffe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c688cae3b60e462e88df69383fe0ffe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39687 } } }
I20260927 18:50:12.393563 20660 sys_catalog.cc:455] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c688cae3b60e462e88df69383fe0ffe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c688cae3b60e462e88df69383fe0ffe8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39687 } } }
I20260927 18:50:12.393769 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:12.393836 20660 sys_catalog.cc:458] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:12.396564 20667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:12.402415 20667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:12.412118 20667 catalog_manager.cc:1347] Generated new cluster ID: e8cf4c449dbe44d8960826fce17b1bc1
I20260927 18:50:12.412262 20667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:12.436229 20667 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:12.438249 20667 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:12.444931 20667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c688cae3b60e462e88df69383fe0ffe8: Generated new TSK 0
I20260927 18:50:12.445904 20667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:12.461927 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20582
W20260927 18:50:12.463207 20644 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:41039
W20260927 18:50:12.464123 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:12.464298 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20582
W20260927 18:50:12.478910 20567 connection.cc:535] client connection to 127.0.0.1:39687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39687): Connection reset by peer (error 104)
2026-09-27T18:50:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:12 dist-test-slave-rfd5 krb5kdc[20680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:12 dist-test-slave-rfd5 krb5kdc[20680](info): set up 2 sockets
Sep 27 18:50:12 dist-test-slave-rfd5 krb5kdc[20680](info): commencing operation
krb5kdc: starting...
W20260927 18:50:13.573633 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:13 dist-test-slave-rfd5 krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:13.840474 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:14.188719 20693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:14.189252 20693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:14.189419 20693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:14.197377 20693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:14.197515 20693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:14.197623 20693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:14.197731 20693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:14.197815 20693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:14.208225 20693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:14.209724 20693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:14.211097 20693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:14.223501 20698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:14.223522 20699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:14.223697 20701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:14.224352 20693 server_base.cc:1034] running on GCE node
I20260927 18:50:14.225289 20693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:14.226751 20693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:14.227963 20693 hybrid_clock.cc:648] HybridClock initialized: now 1790535014227908 us; error 42 us; skew 500 ppm
I20260927 18:50:14.228302 20693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:14 dist-test-slave-rfd5 krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:14.239524 20693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:14.241360 20693 webserver.cc:460] Webserver started at http://127.0.0.1:37911/ using document root <none> and password file <none>
I20260927 18:50:14.242125 20693 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:14.242288 20693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:14.242615 20693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:14.245867 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "2c59dabfc046472fabe0cc7d7e0532ad"
format_stamp: "Formatted at 2026-09-27 18:50:14 on dist-test-slave-rfd5"
server_key: "2c484714d6336e56c65bba80ba40b3b7"
server_key_iv: "4d23386e46044d2a4dd1af9b4c1ab9db"
server_key_version: "encryptionkey@0"
I20260927 18:50:14.246719 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "2c59dabfc046472fabe0cc7d7e0532ad"
format_stamp: "Formatted at 2026-09-27 18:50:14 on dist-test-slave-rfd5"
server_key: "2c484714d6336e56c65bba80ba40b3b7"
server_key_iv: "4d23386e46044d2a4dd1af9b4c1ab9db"
server_key_version: "encryptionkey@0"
I20260927 18:50:14.252509 20693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:50:14.256693 20708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:14.258409 20693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:14.258675 20693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "2c59dabfc046472fabe0cc7d7e0532ad"
format_stamp: "Formatted at 2026-09-27 18:50:14 on dist-test-slave-rfd5"
server_key: "2c484714d6336e56c65bba80ba40b3b7"
server_key_iv: "4d23386e46044d2a4dd1af9b4c1ab9db"
server_key_version: "encryptionkey@0"
I20260927 18:50:14.259033 20693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:14.265528 20693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:14.270081 20693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:14.270427 20693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:14.290547 20693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44373
I20260927 18:50:14.290586 20759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44373 every 8 connection(s)
I20260927 18:50:14.292074 20693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:14.296118 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:14.297636 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20693
I20260927 18:50:14.297847 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:14.298431 31851 external_mini_cluster.cc:1428] Setting key 06626d3efc19447cec7190aa906a999d
I20260927 18:50:14.306502 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:14.300357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:44657 (local address 127.0.0.1:44373)
0927 18:50:14.301252 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:14.301265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:14.301309 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:50:14.302918 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:14.302946 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:14.303337 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:14.303696 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:14.304825 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:14.304831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:14.304845 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:14.304860 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:14.304921 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:14.304950 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:14.305214 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:14.305322 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 18:50:14.305566 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":402,"threads_started":1}
I20260927 18:50:14.310145 20760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad: Bootstrap starting.
I20260927 18:50:14.310416 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:14.314352 20760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:14.315908 20760 log.cc:824] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:14.318919 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:14.316143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:37823 (local address 127.0.0.1:44373)
0927 18:50:14.316342 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:14.316346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:14.316792 (+ 446us) server_negotiation.cc:407] Connection header received
0927 18:50:14.316963 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:14.316971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:14.317039 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:14.317196 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:14.318143 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:14.318147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:14.318150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:14.318165 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:14.318206 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:14.318221 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:14.318316 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:14.318521 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 18:50:14.318628 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:50:14.319825 20760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad: No bootstrap required, opened a new log
I20260927 18:50:14.327436 20760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260927 18:50:14.327924 20760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:14.328045 20760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c59dabfc046472fabe0cc7d7e0532ad, State: Initialized, Role: FOLLOWER
I20260927 18:50:14.328683 20760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260927 18:50:14.328970 20760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:14.329149 20760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:14.329337 20760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:14.331689 20760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260927 18:50:14.332346 20760 leader_election.cc:302] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c59dabfc046472fabe0cc7d7e0532ad; no voters:
I20260927 18:50:14.332976 20760 leader_election.cc:288] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:14.333153 20770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:14.333914 20770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [term 1 LEADER]: Becoming Leader. State: Replica: 2c59dabfc046472fabe0cc7d7e0532ad, State: Running, Role: LEADER
I20260927 18:50:14.334509 20770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260927 18:50:14.335381 20760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:14.337424 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } } }
I20260927 18:50:14.337796 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:14.338521 20772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c59dabfc046472fabe0cc7d7e0532ad. Latest consensus state: current_term: 1 leader_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c59dabfc046472fabe0cc7d7e0532ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } } }
I20260927 18:50:14.339082 20772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:14.340317 20777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:14.346611 20777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:14.356385 20777 catalog_manager.cc:1347] Generated new cluster ID: 98db62bb3a0742c7919d42465b857d91
I20260927 18:50:14.356518 20777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:14.373703 20777 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:14.374233 20777 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:14.384161 20777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c59dabfc046472fabe0cc7d7e0532ad: Generated new TSK 0
I20260927 18:50:14.385471 20777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:14.401011 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20693
W20260927 18:50:14.402314 20755 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:44657
W20260927 18:50:14.403157 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20693: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:14.403326 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20693
2026-09-27T18:50:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:14 dist-test-slave-rfd5 krb5kdc[20791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:14 dist-test-slave-rfd5 krb5kdc[20791](info): set up 2 sockets
Sep 27 18:50:14 dist-test-slave-rfd5 krb5kdc[20791](info): commencing operation
krb5kdc: starting...
W20260927 18:50:15.521621 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:15 dist-test-slave-rfd5 krb5kdc[20791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:15.794560 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:16.146235 20804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:16.146564 20804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:16.146725 20804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:16.154528 20804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:16.154659 20804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:16.154757 20804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:16.154847 20804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:16.154914 20804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:16.165686 20804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:16.167146 20804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:16.168567 20804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:16.181753 20809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:16.181728 20812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:16.181728 20810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:16.182621 20804 server_base.cc:1034] running on GCE node
I20260927 18:50:16.183498 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:16.185060 20804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:16.186376 20804 hybrid_clock.cc:648] HybridClock initialized: now 1790535016186295 us; error 68 us; skew 500 ppm
I20260927 18:50:16.186797 20804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:16 dist-test-slave-rfd5 krb5kdc[20791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:16.190346 20804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:16.191872 20804 webserver.cc:460] Webserver started at http://127.0.0.1:35441/ using document root <none> and password file <none>
I20260927 18:50:16.192552 20804 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:16.192655 20804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:16.192919 20804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:16.196136 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "ef0b62204b9a4a49ab704a3792acae94"
format_stamp: "Formatted at 2026-09-27 18:50:16 on dist-test-slave-rfd5"
server_key: "b811005a92818a10e5b134adafc06f53"
server_key_iv: "b55257a307b4a3ad8d8fb7fd1f54cc7f"
server_key_version: "encryptionkey@0"
I20260927 18:50:16.196902 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "ef0b62204b9a4a49ab704a3792acae94"
format_stamp: "Formatted at 2026-09-27 18:50:16 on dist-test-slave-rfd5"
server_key: "b811005a92818a10e5b134adafc06f53"
server_key_iv: "b55257a307b4a3ad8d8fb7fd1f54cc7f"
server_key_version: "encryptionkey@0"
I20260927 18:50:16.202572 20804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:50:16.206588 20819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:16.208254 20804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:16.208479 20804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "ef0b62204b9a4a49ab704a3792acae94"
format_stamp: "Formatted at 2026-09-27 18:50:16 on dist-test-slave-rfd5"
server_key: "b811005a92818a10e5b134adafc06f53"
server_key_iv: "b55257a307b4a3ad8d8fb7fd1f54cc7f"
server_key_version: "encryptionkey@0"
I20260927 18:50:16.208693 20804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:16.228672 20804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:16.233773 20804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:16.234114 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:16.254827 20804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38551
I20260927 18:50:16.254850 20870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38551 every 8 connection(s)
I20260927 18:50:16.256400 20804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:16.260479 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:16.261878 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20804
I20260927 18:50:16.262107 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:16.262578 31851 external_mini_cluster.cc:1428] Setting key 923b2a70b8aba03acf9b1e8785ea4579
I20260927 18:50:16.276475 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94: Bootstrap starting.
Sep 27 18:50:16 dist-test-slave-rfd5 krb5kdc[20791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:50:16.281157 20871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:16.282641 20871 log.cc:824] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:16.286731 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94: No bootstrap required, opened a new log
I20260927 18:50:16.290916 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:16.264653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:54781 (local address 127.0.0.1:38551)
0927 18:50:16.265549 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:16.265572 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:16.265621 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:50:16.267346 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:16.267377 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:16.267878 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:16.268258 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:16.270278 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:16.272323 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:16.274352 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:16.275166 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:16.275352 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:16.281282 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:16.281292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:16.281333 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:16.281405 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:16.285504 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:16.286732 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:16.286746 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:16.286753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:16.287050 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:16.287686 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:16.287694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:16.287697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:16.288341 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:16.288615 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:16.289348 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 18:50:16.289779 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":420,"threads_started":1}
I20260927 18:50:16.295764 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:16.296658 20871 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0b62204b9a4a49ab704a3792acae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260927 18:50:16.297091 20871 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:16.297261 20871 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0b62204b9a4a49ab704a3792acae94, State: Initialized, Role: FOLLOWER
I20260927 18:50:16.297896 20871 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0b62204b9a4a49ab704a3792acae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260927 18:50:16.298168 20871 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:16.298323 20871 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:16.298499 20871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:16.300838 20871 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0b62204b9a4a49ab704a3792acae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260927 18:50:16.301544 20871 leader_election.cc:302] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef0b62204b9a4a49ab704a3792acae94; no voters:
I20260927 18:50:16.302290 20871 leader_election.cc:288] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:16.302416 20880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:16.304312 20880 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [term 1 LEADER]: Becoming Leader. State: Replica: ef0b62204b9a4a49ab704a3792acae94, State: Running, Role: LEADER
I20260927 18:50:16.304941 20880 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0b62204b9a4a49ab704a3792acae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260927 18:50:16.305171 20871 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:16.309659 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef0b62204b9a4a49ab704a3792acae94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0b62204b9a4a49ab704a3792acae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } } }
I20260927 18:50:16.310113 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:16.310662 20882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef0b62204b9a4a49ab704a3792acae94. Latest consensus state: current_term: 1 leader_uuid: "ef0b62204b9a4a49ab704a3792acae94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef0b62204b9a4a49ab704a3792acae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } } }
I20260927 18:50:16.310957 20882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:16.315021 20887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:16.322036 20887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:16.325734 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:16.303601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:53377 (local address 127.0.0.1:38551)
0927 18:50:16.303835 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:16.303844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:16.305325 (+ 1481us) server_negotiation.cc:407] Connection header received
0927 18:50:16.305607 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:16.305613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:16.305682 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:16.305810 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:16.307549 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:16.309029 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:16.311203 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:16.312082 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:16.312297 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:16.316055 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:16.316063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:16.316067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:16.316127 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:16.320692 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:16.321967 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:16.321974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:16.321977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:16.322280 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:16.322948 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:16.322955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:16.322961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:16.323405 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:16.323665 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:16.325063 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0927 18:50:16.325330 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:50:16.336238 20887 catalog_manager.cc:1347] Generated new cluster ID: 969afd423bab43aca351d1a6c4a400f8
I20260927 18:50:16.336382 20887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:16.359584 20887 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:16.361456 20887 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:16.373225 20887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef0b62204b9a4a49ab704a3792acae94: Generated new TSK 0
I20260927 18:50:16.374445 20887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:16.384974 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20804
I20260927 18:50:16.386361 20866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:16.480255 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20804
W20260927 18:50:16.497825 20789 connection.cc:535] client connection to 127.0.0.1:38551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38551): Connection reset by peer (error 104)
2026-09-27T18:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:16 dist-test-slave-rfd5 krb5kdc[20903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:16 dist-test-slave-rfd5 krb5kdc[20903](info): set up 2 sockets
Sep 27 18:50:16 dist-test-slave-rfd5 krb5kdc[20903](info): commencing operation
krb5kdc: starting...
W20260927 18:50:17.605665 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:17 dist-test-slave-rfd5 krb5kdc[20903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:17.877017 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:18.228576 20916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:18.228910 20916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:18.229154 20916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:18.236812 20916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:18.236934 20916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:18.237066 20916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:18.237190 20916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:18.237254 20916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:18.247665 20916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:18.248991 20916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:18.250430 20916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:18.263336 20922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:18.263408 20924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:18.263367 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:18.264599 20916 server_base.cc:1034] running on GCE node
I20260927 18:50:18.265605 20916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:18.267249 20916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:18.268471 20916 hybrid_clock.cc:648] HybridClock initialized: now 1790535018268402 us; error 66 us; skew 500 ppm
I20260927 18:50:18.268831 20916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:18 dist-test-slave-rfd5 krb5kdc[20903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:18.272262 20916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:18.273976 20916 webserver.cc:460] Webserver started at http://127.0.0.1:33077/ using document root <none> and password file <none>
I20260927 18:50:18.274756 20916 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:18.274904 20916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:18.275238 20916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:18.278616 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "6c4004b0129a45118e29896af953c699"
format_stamp: "Formatted at 2026-09-27 18:50:18 on dist-test-slave-rfd5"
server_key: "7b81531f9667464fcebd9e6c05d08563"
server_key_iv: "1a0e9dd7654038da737703aad4760560"
server_key_version: "encryptionkey@0"
I20260927 18:50:18.279498 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "6c4004b0129a45118e29896af953c699"
format_stamp: "Formatted at 2026-09-27 18:50:18 on dist-test-slave-rfd5"
server_key: "7b81531f9667464fcebd9e6c05d08563"
server_key_iv: "1a0e9dd7654038da737703aad4760560"
server_key_version: "encryptionkey@0"
I20260927 18:50:18.285321 20916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:50:18.289345 20931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:18.290917 20916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:18.291132 20916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "6c4004b0129a45118e29896af953c699"
format_stamp: "Formatted at 2026-09-27 18:50:18 on dist-test-slave-rfd5"
server_key: "7b81531f9667464fcebd9e6c05d08563"
server_key_iv: "1a0e9dd7654038da737703aad4760560"
server_key_version: "encryptionkey@0"
I20260927 18:50:18.291369 20916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:18.304873 20916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:18.309461 20916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:18.309767 20916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:18.329221 20916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41369
I20260927 18:50:18.329272 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41369 every 8 connection(s)
I20260927 18:50:18.330703 20916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:18.334555 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:18.338562 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 20916
I20260927 18:50:18.338761 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:18.339198 31851 external_mini_cluster.cc:1428] Setting key 51ab7935bc4d6c65e497b4462ffaaf49
I20260927 18:50:18.348479 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699: Bootstrap starting.
I20260927 18:50:18.352447 20983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:18.353817 20983 log.cc:824] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:18.354132 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:18.341190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:48077 (local address 127.0.0.1:41369)
0927 18:50:18.341980 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:18.341997 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:18.342047 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:50:18.343805 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:18.343836 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:18.344416 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:18.344891 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:18.345439 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:18.347594 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:18.348956 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:18.349974 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:18.350189 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:18.351344 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:18.351355 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:18.351372 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:18.352077 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:18.352179 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:18.352215 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:18.352387 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:18.352598 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:50:18.352805 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":317,"threads_started":1}
I20260927 18:50:18.357421 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699: No bootstrap required, opened a new log
I20260927 18:50:18.359540 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:18.366648 20983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4004b0129a45118e29896af953c699" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260927 18:50:18.367116 20983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:18.367225 20983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4004b0129a45118e29896af953c699, State: Initialized, Role: FOLLOWER
I20260927 18:50:18.367925 20983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4004b0129a45118e29896af953c699" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260927 18:50:18.368227 20983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:18.368379 20983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:18.368564 20983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:18.371034 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:18.363986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:39487 (local address 127.0.0.1:41369)
0927 18:50:18.364177 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:18.364184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:18.364537 (+ 353us) server_negotiation.cc:407] Connection header received
0927 18:50:18.364692 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:18.364697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:18.364793 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:18.364911 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:18.365436 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:18.366518 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:18.367977 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:18.368678 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:18.368810 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:18.370336 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:18.370341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:18.370348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:18.370367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:18.370430 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:18.370445 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:18.370542 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:18.370754 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 18:50:18.370801 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:50:18.371923 20983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4004b0129a45118e29896af953c699" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260927 18:50:18.372540 20983 leader_election.cc:302] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c4004b0129a45118e29896af953c699; no voters:
I20260927 18:50:18.373186 20983 leader_election.cc:288] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:18.373332 20993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:18.374035 20993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [term 1 LEADER]: Becoming Leader. State: Replica: 6c4004b0129a45118e29896af953c699, State: Running, Role: LEADER
I20260927 18:50:18.374567 20993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4004b0129a45118e29896af953c699" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260927 18:50:18.375382 20983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:18.377453 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c4004b0129a45118e29896af953c699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4004b0129a45118e29896af953c699" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260927 18:50:18.377444 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c4004b0129a45118e29896af953c699. Latest consensus state: current_term: 1 leader_uuid: "6c4004b0129a45118e29896af953c699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4004b0129a45118e29896af953c699" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260927 18:50:18.377908 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:18.377908 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:18.382082 20998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:18.387401 20998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:18.398231 20998 catalog_manager.cc:1347] Generated new cluster ID: 20961505f924499b8e169c5ba83b0b86
I20260927 18:50:18.398404 20998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:18.425320 20998 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:18.427402 20998 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:18.438447 20998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c4004b0129a45118e29896af953c699: Generated new TSK 0
I20260927 18:50:18.439357 20998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:18.452289 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20916
W20260927 18:50:18.453507 20978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:48077
W20260927 18:50:18.454372 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 20916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:18.454535 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 20916
W20260927 18:50:18.468654 20901 connection.cc:535] client connection to 127.0.0.1:41369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41369): Connection reset by peer (error 104)
2026-09-27T18:50:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:18 dist-test-slave-rfd5 krb5kdc[21014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:18 dist-test-slave-rfd5 krb5kdc[21014](info): set up 2 sockets
Sep 27 18:50:18 dist-test-slave-rfd5 krb5kdc[21014](info): commencing operation
krb5kdc: starting...
W20260927 18:50:19.569813 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:19 dist-test-slave-rfd5 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:19.849152 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:20.199731 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:20.200098 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:20.200278 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:20.207646 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:20.207834 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:20.207952 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:20.208060 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:20.208148 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:20.218403 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:20.219852 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:20.221321 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:20.233999 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:20.234112 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:20.234046 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:20.234973 21027 server_base.cc:1034] running on GCE node
I20260927 18:50:20.235798 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:20.237485 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:20.238703 21027 hybrid_clock.cc:648] HybridClock initialized: now 1790535020238660 us; error 33 us; skew 500 ppm
I20260927 18:50:20.239033 21027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:20 dist-test-slave-rfd5 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:20.242341 21027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:20.243944 21027 webserver.cc:460] Webserver started at http://127.0.0.1:33461/ using document root <none> and password file <none>
I20260927 18:50:20.244668 21027 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:20.244778 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:20.245044 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:20.248327 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "30ffda58271c4db28cdd5bb63c02862d"
format_stamp: "Formatted at 2026-09-27 18:50:20 on dist-test-slave-rfd5"
server_key: "91db9e30bd18547ae3d7b02c8906ce3b"
server_key_iv: "f6cc30cace15d748c33acb3eaca313ed"
server_key_version: "encryptionkey@0"
I20260927 18:50:20.249269 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "30ffda58271c4db28cdd5bb63c02862d"
format_stamp: "Formatted at 2026-09-27 18:50:20 on dist-test-slave-rfd5"
server_key: "91db9e30bd18547ae3d7b02c8906ce3b"
server_key_iv: "f6cc30cace15d748c33acb3eaca313ed"
server_key_version: "encryptionkey@0"
I20260927 18:50:20.255143 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:20.259056 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:20.260747 21027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:50:20.260936 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "30ffda58271c4db28cdd5bb63c02862d"
format_stamp: "Formatted at 2026-09-27 18:50:20 on dist-test-slave-rfd5"
server_key: "91db9e30bd18547ae3d7b02c8906ce3b"
server_key_iv: "f6cc30cace15d748c33acb3eaca313ed"
server_key_version: "encryptionkey@0"
I20260927 18:50:20.261230 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:20.267746 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:20.272552 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:20.272826 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:20.292397 21027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35217
I20260927 18:50:20.292451 21093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35217 every 8 connection(s)
I20260927 18:50:20.294075 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:20.295270 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21027
I20260927 18:50:20.295455 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:20.295919 31851 external_mini_cluster.cc:1428] Setting key bbf1b41a97327e50c9fd9a06a32ce411
I20260927 18:50:20.299743 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:20.306819 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:20.297856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:55281 (local address 127.0.0.1:35217)
0927 18:50:20.298715 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:20.298728 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:20.300630 (+ 1902us) server_negotiation.cc:407] Connection header received
0927 18:50:20.302704 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:20.302729 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:20.303162 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:20.303648 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:20.304875 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:20.304882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:20.304901 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:20.304922 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:20.305021 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:20.305050 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:20.305375 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:20.305548 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:50:20.305772 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":381,"threads_started":1}
I20260927 18:50:20.311131 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:20.312774 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d: Bootstrap starting.
I20260927 18:50:20.316388 21094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:20.317662 21094 log.cc:824] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:20.318989 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:20.315801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:48489 (local address 127.0.0.1:35217)
0927 18:50:20.316045 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:20.316056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:20.316651 (+ 595us) server_negotiation.cc:407] Connection header received
0927 18:50:20.316865 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:20.316876 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:20.316927 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:20.317037 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:20.318104 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:20.318110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:20.318121 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:20.318137 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:20.318209 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:20.318240 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:20.318381 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:20.318570 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:50:20.318652 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:50:20.321269 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d: No bootstrap required, opened a new log
I20260927 18:50:20.329299 21094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffda58271c4db28cdd5bb63c02862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 18:50:20.329890 21094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:20.330041 21094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ffda58271c4db28cdd5bb63c02862d, State: Initialized, Role: FOLLOWER
I20260927 18:50:20.330789 21094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffda58271c4db28cdd5bb63c02862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 18:50:20.331007 21094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:20.331110 21094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:20.331251 21094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:20.333566 21094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffda58271c4db28cdd5bb63c02862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 18:50:20.334084 21094 leader_election.cc:302] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ffda58271c4db28cdd5bb63c02862d; no voters:
I20260927 18:50:20.334677 21094 leader_election.cc:288] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:20.334867 21104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:20.335983 21104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [term 1 LEADER]: Becoming Leader. State: Replica: 30ffda58271c4db28cdd5bb63c02862d, State: Running, Role: LEADER
I20260927 18:50:20.336527 21104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffda58271c4db28cdd5bb63c02862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260927 18:50:20.336854 21094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:20.339764 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ffda58271c4db28cdd5bb63c02862d. Latest consensus state: current_term: 1 leader_uuid: "30ffda58271c4db28cdd5bb63c02862d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffda58271c4db28cdd5bb63c02862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260927 18:50:20.340276 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:20.340610 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ffda58271c4db28cdd5bb63c02862d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ffda58271c4db28cdd5bb63c02862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260927 18:50:20.340917 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:20.341817 21113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:20.347720 21113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:20.357239 21113 catalog_manager.cc:1347] Generated new cluster ID: ea22f1b0bfff43ad9e13b7e2433828b1
I20260927 18:50:20.357381 21113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:20.379069 21113 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:20.379662 21113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:20.389330 21113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ffda58271c4db28cdd5bb63c02862d: Generated new TSK 0
I20260927 18:50:20.390277 21113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:20.400151 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21027
W20260927 18:50:20.401561 21089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:55281
W20260927 18:50:20.402384 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:20.402561 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21027
2026-09-27T18:50:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:20 dist-test-slave-rfd5 krb5kdc[21125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:20 dist-test-slave-rfd5 krb5kdc[21125](info): set up 2 sockets
Sep 27 18:50:20 dist-test-slave-rfd5 krb5kdc[21125](info): commencing operation
krb5kdc: starting...
W20260927 18:50:21.509668 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:21 dist-test-slave-rfd5 krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:21.774729 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:43195
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:43195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:22.117280 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:22.117771 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:22.117947 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:22.126745 21138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:22.126881 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:22.126972 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:22.127089 21138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:22.127161 21138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:22.137763 21138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:43195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:22.139163 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:22.140532 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:22.152637 21143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:22.152588 21144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:22.152551 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:22.153409 21138 server_base.cc:1034] running on GCE node
I20260927 18:50:22.154354 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:22.155911 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:22.157199 21138 hybrid_clock.cc:648] HybridClock initialized: now 1790535022157135 us; error 48 us; skew 500 ppm
I20260927 18:50:22.157573 21138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:22 dist-test-slave-rfd5 krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535022, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:22.160743 21138 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:22.162412 21138 webserver.cc:460] Webserver started at http://127.31.26.254:33421/ using document root <none> and password file <none>
I20260927 18:50:22.163164 21138 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:22.163324 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:22.163641 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:22.166826 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "92e97bec6b2d40aba571f90321e121e8"
format_stamp: "Formatted at 2026-09-27 18:50:22 on dist-test-slave-rfd5"
server_key: "7a4f4ea03ea826a387bb272835b09784"
server_key_iv: "19e7db85bf86638fd0a92bf13e59507e"
server_key_version: "encryptionkey@0"
I20260927 18:50:22.167650 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "92e97bec6b2d40aba571f90321e121e8"
format_stamp: "Formatted at 2026-09-27 18:50:22 on dist-test-slave-rfd5"
server_key: "7a4f4ea03ea826a387bb272835b09784"
server_key_iv: "19e7db85bf86638fd0a92bf13e59507e"
server_key_version: "encryptionkey@0"
I20260927 18:50:22.173280 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 18:50:22.176855 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:22.178368 21138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:22.178579 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "92e97bec6b2d40aba571f90321e121e8"
format_stamp: "Formatted at 2026-09-27 18:50:22 on dist-test-slave-rfd5"
server_key: "7a4f4ea03ea826a387bb272835b09784"
server_key_iv: "19e7db85bf86638fd0a92bf13e59507e"
server_key_version: "encryptionkey@0"
I20260927 18:50:22.178803 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:22.196558 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:22.201442 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:22.201772 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:22.222307 21138 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:43195
I20260927 18:50:22.222314 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:43195 every 8 connection(s)
I20260927 18:50:22.223785 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:22.227646 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:22.230942 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21138
I20260927 18:50:22.231142 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:22.231698 31851 external_mini_cluster.cc:1428] Setting key 5065648a14820c89ad910d021f9abdae
I20260927 18:50:22.242282 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8: Bootstrap starting.
I20260927 18:50:22.246150 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:22.247231 21205 log.cc:824] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:50:22 dist-test-slave-rfd5 krb5kdc[21125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:22.250564 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8: No bootstrap required, opened a new log
I20260927 18:50:22.257678 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:22.233983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48153 (local address 127.31.26.254:43195)
0927 18:50:22.234954 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:22.234971 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:22.235443 (+ 472us) server_negotiation.cc:407] Connection header received
0927 18:50:22.237795 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:22.237827 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:22.238442 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:22.238933 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:22.240209 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:22.242206 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:22.243540 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:22.244682 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:22.244870 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:22.249436 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:22.249444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:22.249460 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:22.249504 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:22.252860 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:22.253839 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:22.253861 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:22.253868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:22.254055 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:22.254573 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:22.254577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:22.254579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:22.255198 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:22.255582 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:22.256112 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 18:50:22.256479 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":404,"threads_started":1}
I20260927 18:50:22.259383 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e97bec6b2d40aba571f90321e121e8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 43195 } }
I20260927 18:50:22.259816 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:22.259943 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e97bec6b2d40aba571f90321e121e8, State: Initialized, Role: FOLLOWER
I20260927 18:50:22.260625 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e97bec6b2d40aba571f90321e121e8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 43195 } }
I20260927 18:50:22.260931 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:22.261085 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:22.261298 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:22.262367 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:22.263820 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e97bec6b2d40aba571f90321e121e8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 43195 } }
I20260927 18:50:22.264364 21205 leader_election.cc:302] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e97bec6b2d40aba571f90321e121e8; no voters:
I20260927 18:50:22.265033 21205 leader_election.cc:288] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:22.265187 21211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:22.267808 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:22.269449 21211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [term 1 LEADER]: Becoming Leader. State: Replica: 92e97bec6b2d40aba571f90321e121e8, State: Running, Role: LEADER
I20260927 18:50:22.270329 21211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e97bec6b2d40aba571f90321e121e8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 43195 } }
I20260927 18:50:22.274150 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e97bec6b2d40aba571f90321e121e8. Latest consensus state: current_term: 1 leader_uuid: "92e97bec6b2d40aba571f90321e121e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e97bec6b2d40aba571f90321e121e8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 43195 } } }
I20260927 18:50:22.274837 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:22.275650 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e97bec6b2d40aba571f90321e121e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e97bec6b2d40aba571f90321e121e8" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 43195 } } }
I20260927 18:50:22.276126 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:22.281705 21220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:22.289417 21220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:22.297734 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:22.277363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47819 (local address 127.31.26.254:43195)
0927 18:50:22.277571 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:22.277578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:22.278788 (+ 1210us) server_negotiation.cc:407] Connection header received
0927 18:50:22.279126 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:22.279135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:22.279242 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:22.279335 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:22.281869 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:22.283061 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:22.285413 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:22.286114 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:22.286230 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:22.289941 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:22.289949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:22.289953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:22.289996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:22.293944 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:22.295036 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:22.295042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:22.295045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:22.295202 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:22.295770 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:22.295775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:22.295778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:22.296167 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:22.296417 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:22.297152 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 18:50:22.297354 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:50:22.304052 21220 catalog_manager.cc:1347] Generated new cluster ID: 8b71dfd58f7c4d359bd08938d9e57d61
I20260927 18:50:22.304193 21220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:22.357065 21220 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:22.358794 21220 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:22.375973 21220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e97bec6b2d40aba571f90321e121e8: Generated new TSK 0
I20260927 18:50:22.377140 21220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:22.392763 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21138
I20260927 18:50:22.394160 21200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:22.492954 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21138
W20260927 18:50:22.511277 21123 connection.cc:535] client connection to 127.31.26.254:43195 recv error: Network error: failed to read from TLS socket (remote: 127.31.26.254:43195): Connection reset by peer (error 104)
2026-09-27T18:50:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:22 dist-test-slave-rfd5 krb5kdc[21237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:22 dist-test-slave-rfd5 krb5kdc[21237](info): set up 2 sockets
Sep 27 18:50:22 dist-test-slave-rfd5 krb5kdc[21237](info): commencing operation
krb5kdc: starting...
W20260927 18:50:23.609735 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:23 dist-test-slave-rfd5 krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:23.876235 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:45465
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:45465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:24.225072 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:24.225432 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:24.225610 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:24.233384 21250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:24.233520 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:24.233645 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:24.233771 21250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:24.233852 21250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:24.244376 21250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:45465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:45465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:24.245855 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:24.247251 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:24.260357 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:24.260315 21258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:24.260284 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:24.261344 21250 server_base.cc:1034] running on GCE node
I20260927 18:50:24.262218 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:24.263813 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:24.265151 21250 hybrid_clock.cc:648] HybridClock initialized: now 1790535024265028 us; error 52 us; skew 500 ppm
I20260927 18:50:24.265556 21250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:24 dist-test-slave-rfd5 krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535024, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:24.269013 21250 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:24.270697 21250 webserver.cc:460] Webserver started at http://127.31.26.254:39107/ using document root <none> and password file <none>
I20260927 18:50:24.271457 21250 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:24.271620 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:24.271944 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:24.275269 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "f94a5739e9cb47bc8519a1742f2bff6e"
format_stamp: "Formatted at 2026-09-27 18:50:24 on dist-test-slave-rfd5"
server_key: "e4b849b470611930b938a7a7ccab9e08"
server_key_iv: "b902070cbb6cd55f168d0850f77ee571"
server_key_version: "encryptionkey@0"
I20260927 18:50:24.276119 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "f94a5739e9cb47bc8519a1742f2bff6e"
format_stamp: "Formatted at 2026-09-27 18:50:24 on dist-test-slave-rfd5"
server_key: "e4b849b470611930b938a7a7ccab9e08"
server_key_iv: "b902070cbb6cd55f168d0850f77ee571"
server_key_version: "encryptionkey@0"
I20260927 18:50:24.281785 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:50:24.285517 21265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:24.286980 21250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:50:24.287194 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "f94a5739e9cb47bc8519a1742f2bff6e"
format_stamp: "Formatted at 2026-09-27 18:50:24 on dist-test-slave-rfd5"
server_key: "e4b849b470611930b938a7a7ccab9e08"
server_key_iv: "b902070cbb6cd55f168d0850f77ee571"
server_key_version: "encryptionkey@0"
I20260927 18:50:24.287423 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:24.305838 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:24.310525 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:24.310865 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:24.330405 21250 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:45465
I20260927 18:50:24.330427 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:45465 every 8 connection(s)
I20260927 18:50:24.331948 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:24.333338 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21250
I20260927 18:50:24.333534 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:24.334064 31851 external_mini_cluster.cc:1428] Setting key ce92639e5a4b331a93128d8de681b422
I20260927 18:50:24.337064 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:24.347617 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:24.336012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34393 (local address 127.31.26.254:45465)
0927 18:50:24.336819 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:24.336831 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:24.336874 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:50:24.338666 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:24.338689 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:24.339101 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:24.339528 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:24.340124 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:24.341526 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:24.342861 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:24.343556 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:24.343757 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:24.345153 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:24.345165 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:24.345193 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:24.345820 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:24.345883 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:24.345915 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:24.346109 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:24.346401 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 18:50:24.346608 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":362,"threads_started":1}
I20260927 18:50:24.348740 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e: Bootstrap starting.
I20260927 18:50:24.351869 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:24.352387 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:24.353505 21317 log.cc:824] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:24.356835 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e: No bootstrap required, opened a new log
I20260927 18:50:24.363682 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:24.356639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56957 (local address 127.31.26.254:45465)
0927 18:50:24.356875 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:24.356883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:24.357405 (+ 522us) server_negotiation.cc:407] Connection header received
0927 18:50:24.357543 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:24.357548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:24.357633 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:24.357771 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:24.358266 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:24.359485 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:24.360808 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:24.361534 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:24.361660 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:24.362849 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:24.362856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:24.362861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:24.362888 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:24.362946 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:24.362972 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:24.363093 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:24.363272 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 18:50:24.363343 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:50:24.366897 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45465 } }
I20260927 18:50:24.367429 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:24.367548 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94a5739e9cb47bc8519a1742f2bff6e, State: Initialized, Role: FOLLOWER
I20260927 18:50:24.368233 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45465 } }
I20260927 18:50:24.368458 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:24.368563 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:24.368690 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:24.370959 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45465 } }
I20260927 18:50:24.371505 21317 leader_election.cc:302] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f94a5739e9cb47bc8519a1742f2bff6e; no voters:
I20260927 18:50:24.372131 21317 leader_election.cc:288] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:24.372326 21327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:24.373788 21327 raft_consensus.cc:686] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [term 1 LEADER]: Becoming Leader. State: Replica: f94a5739e9cb47bc8519a1742f2bff6e, State: Running, Role: LEADER
I20260927 18:50:24.374583 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:24.374549 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45465 } }
I20260927 18:50:24.377671 21329 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f94a5739e9cb47bc8519a1742f2bff6e. Latest consensus state: current_term: 1 leader_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45465 } } }
I20260927 18:50:24.378552 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94a5739e9cb47bc8519a1742f2bff6e" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 45465 } } }
I20260927 18:50:24.378818 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:24.378090 21329 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:24.379617 21336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:24.387678 21336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:24.398511 21336 catalog_manager.cc:1347] Generated new cluster ID: 155b14d33e234d26bb0fef5451a1d064
I20260927 18:50:24.398666 21336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:24.412381 21336 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:24.413954 21336 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:24.427423 21336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f94a5739e9cb47bc8519a1742f2bff6e: Generated new TSK 0
I20260927 18:50:24.428598 21336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:24.445133 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21250
W20260927 18:50:24.446973 21312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34393
W20260927 18:50:24.447901 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:24.448100 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21250
W20260927 18:50:24.463065 21235 connection.cc:535] client connection to 127.31.26.254:45465 recv error: Network error: failed to read from TLS socket (remote: 127.31.26.254:45465): Connection reset by peer (error 104)
2026-09-27T18:50:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:24 dist-test-slave-rfd5 krb5kdc[21348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:24 dist-test-slave-rfd5 krb5kdc[21348](info): set up 2 sockets
Sep 27 18:50:24 dist-test-slave-rfd5 krb5kdc[21348](info): commencing operation
krb5kdc: starting...
W20260927 18:50:25.557705 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:25 dist-test-slave-rfd5 krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:25.824991 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34791
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:26.166735 21361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:26.167111 21361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:26.167297 21361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:26.174800 21361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:26.174974 21361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:26.175096 21361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:26.175201 21361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:26.175292 21361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:26.185648 21361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:34791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:26.187043 21361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:26.188405 21361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:26.200920 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.200947 21367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.200855 21369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:26.201743 21361 server_base.cc:1034] running on GCE node
I20260927 18:50:26.202577 21361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:26.204048 21361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:26.205240 21361 hybrid_clock.cc:648] HybridClock initialized: now 1790535026205206 us; error 36 us; skew 500 ppm
I20260927 18:50:26.205606 21361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:26 dist-test-slave-rfd5 krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535026, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:26.218020 21361 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:26.219766 21361 webserver.cc:460] Webserver started at http://127.31.26.254:40457/ using document root <none> and password file <none>
I20260927 18:50:26.220538 21361 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:26.220693 21361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:26.221032 21361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:26.224336 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "17a7f6a5e2df4efab2243f3391ac7dab"
format_stamp: "Formatted at 2026-09-27 18:50:26 on dist-test-slave-rfd5"
server_key: "39e44c105deea6bc5fe3c8c6a1322769"
server_key_iv: "c10d1594d2cd74ed5d78088294ec402f"
server_key_version: "encryptionkey@0"
I20260927 18:50:26.225265 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "17a7f6a5e2df4efab2243f3391ac7dab"
format_stamp: "Formatted at 2026-09-27 18:50:26 on dist-test-slave-rfd5"
server_key: "39e44c105deea6bc5fe3c8c6a1322769"
server_key_iv: "c10d1594d2cd74ed5d78088294ec402f"
server_key_version: "encryptionkey@0"
I20260927 18:50:26.230883 21361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:26.234709 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.236163 21361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260927 18:50:26.236377 21361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "17a7f6a5e2df4efab2243f3391ac7dab"
format_stamp: "Formatted at 2026-09-27 18:50:26 on dist-test-slave-rfd5"
server_key: "39e44c105deea6bc5fe3c8c6a1322769"
server_key_iv: "c10d1594d2cd74ed5d78088294ec402f"
server_key_version: "encryptionkey@0"
I20260927 18:50:26.236634 21361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:26.242791 21361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:26.247602 21361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:26.247936 21361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:26.268146 21361 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:34791
I20260927 18:50:26.268187 21427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:34791 every 8 connection(s)
I20260927 18:50:26.269834 21361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:26.271694 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21361
I20260927 18:50:26.271889 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:26.272364 31851 external_mini_cluster.cc:1428] Setting key 13ce663a77c48c9675c9e2ec8b180d43
I20260927 18:50:26.274499 21428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:26.280979 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.274237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50197 (local address 127.31.26.254:34791)
0927 18:50:26.275140 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:26.275153 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:26.275199 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:50:26.277062 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:26.277152 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:26.277569 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:26.277968 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:26.279209 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:26.279215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:26.279231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:26.279246 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:26.279330 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:26.279353 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:26.279531 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:26.279665 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:50:26.279910 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":441,"threads_started":1}
I20260927 18:50:26.285298 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:26.287812 21428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab: Bootstrap starting.
I20260927 18:50:26.291638 21428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:26.292665 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.289950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56171 (local address 127.31.26.254:34791)
0927 18:50:26.290146 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:26.290152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:26.290585 (+ 433us) server_negotiation.cc:407] Connection header received
0927 18:50:26.290730 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:26.290736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:26.290809 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:26.290934 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:26.291853 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:26.291858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:26.291869 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:26.291885 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:26.291925 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:26.291941 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:26.292068 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:26.292202 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:50:26.292277 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:50:26.293292 21428 log.cc:824] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:26.296307 21428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab: No bootstrap required, opened a new log
I20260927 18:50:26.303762 21428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34791 } }
I20260927 18:50:26.304185 21428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:26.304297 21428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a7f6a5e2df4efab2243f3391ac7dab, State: Initialized, Role: FOLLOWER
I20260927 18:50:26.304898 21428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34791 } }
I20260927 18:50:26.305169 21428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:26.305325 21428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:26.305505 21428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:26.307744 21428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34791 } }
I20260927 18:50:26.308284 21428 leader_election.cc:302] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a7f6a5e2df4efab2243f3391ac7dab; no voters:
I20260927 18:50:26.309034 21428 leader_election.cc:288] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:26.309177 21438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:26.309842 21438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [term 1 LEADER]: Becoming Leader. State: Replica: 17a7f6a5e2df4efab2243f3391ac7dab, State: Running, Role: LEADER
I20260927 18:50:26.310500 21438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34791 } }
I20260927 18:50:26.311520 21428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:26.313666 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34791 } } }
I20260927 18:50:26.314101 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:26.315609 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a7f6a5e2df4efab2243f3391ac7dab. Latest consensus state: current_term: 1 leader_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a7f6a5e2df4efab2243f3391ac7dab" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34791 } } }
I20260927 18:50:26.315927 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:26.316083 21444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:26.322431 21444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:26.331977 21444 catalog_manager.cc:1347] Generated new cluster ID: 9c2b479824484ec185bbfa8d0956a5d7
I20260927 18:50:26.332115 21444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:26.359447 21444 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:26.360010 21444 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:26.368795 21444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a7f6a5e2df4efab2243f3391ac7dab: Generated new TSK 0
I20260927 18:50:26.369982 21444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:26.382404 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21361
W20260927 18:50:26.383798 21423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50197
W20260927 18:50:26.384657 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21361: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:26.384825 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21361
2026-09-27T18:50:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:26 dist-test-slave-rfd5 krb5kdc[21459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:26 dist-test-slave-rfd5 krb5kdc[21459](info): set up 2 sockets
Sep 27 18:50:26 dist-test-slave-rfd5 krb5kdc[21459](info): commencing operation
krb5kdc: starting...
W20260927 18:50:27.489759 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:27 dist-test-slave-rfd5 krb5kdc[21459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:27.756747 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41807
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:28.100572 21472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:28.100880 21472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:28.100973 21472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:28.108415 21472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:28.108553 21472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:28.108603 21472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:28.108662 21472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:28.108708 21472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:28.122025 21472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:28.123392 21472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:28.124716 21472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:28.137501 21480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:28.137442 21477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:28.137452 21478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:28.138244 21472 server_base.cc:1034] running on GCE node
I20260927 18:50:28.139030 21472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:28.140444 21472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:28.141646 21472 hybrid_clock.cc:648] HybridClock initialized: now 1790535028141612 us; error 34 us; skew 500 ppm
I20260927 18:50:28.141978 21472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:28 dist-test-slave-rfd5 krb5kdc[21459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:28.145272 21472 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:28.146871 21472 webserver.cc:460] Webserver started at http://127.31.26.254:35587/ using document root <none> and password file <none>
I20260927 18:50:28.147568 21472 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:28.147671 21472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:28.147938 21472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:28.151140 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "8f05e97a4c794e40aa63b0e9f4922480"
format_stamp: "Formatted at 2026-09-27 18:50:28 on dist-test-slave-rfd5"
server_key: "c8d1ecf4fe008a5ad53d654e83183901"
server_key_iv: "b73b712d27aab704ac8317764b1700de"
server_key_version: "encryptionkey@0"
I20260927 18:50:28.152004 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "8f05e97a4c794e40aa63b0e9f4922480"
format_stamp: "Formatted at 2026-09-27 18:50:28 on dist-test-slave-rfd5"
server_key: "c8d1ecf4fe008a5ad53d654e83183901"
server_key_iv: "b73b712d27aab704ac8317764b1700de"
server_key_version: "encryptionkey@0"
I20260927 18:50:28.157871 21472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:50:28.161990 21487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:28.163507 21472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:28.163719 21472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "8f05e97a4c794e40aa63b0e9f4922480"
format_stamp: "Formatted at 2026-09-27 18:50:28 on dist-test-slave-rfd5"
server_key: "c8d1ecf4fe008a5ad53d654e83183901"
server_key_iv: "b73b712d27aab704ac8317764b1700de"
server_key_version: "encryptionkey@0"
I20260927 18:50:28.163966 21472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:28.197585 21472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:28.202344 21472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:28.202666 21472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:28.222512 21472 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41807
I20260927 18:50:28.222512 21538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41807 every 8 connection(s)
I20260927 18:50:28.224102 21472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:28.228251 21539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:28.233464 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21472
I20260927 18:50:28.233680 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:28.234148 31851 external_mini_cluster.cc:1428] Setting key e2fbc6ded42aa070ff174f64a932132b
I20260927 18:50:28.240007 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480: Bootstrap starting.
I20260927 18:50:28.243377 21539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:28.244517 21539 log.cc:824] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:28.248414 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480: No bootstrap required, opened a new log
Sep 27 18:50:28 dist-test-slave-rfd5 krb5kdc[21459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:28.260160 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.236026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57671 (local address 127.31.26.254:41807)
0927 18:50:28.236876 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:28.236888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:28.236931 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:50:28.238772 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:28.238798 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:28.239230 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:28.239636 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:28.241160 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.242536 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.243907 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.244630 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.244821 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:28.250064 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:28.250076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:28.250104 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:28.250159 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:28.254643 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.255582 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.255592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.255602 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.255875 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.256415 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.256425 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.256430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.257351 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:28.257681 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:28.258111 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 18:50:28.258653 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":387,"threads_started":1}
I20260927 18:50:28.259876 21539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f05e97a4c794e40aa63b0e9f4922480" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41807 } }
I20260927 18:50:28.260360 21539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:28.260495 21539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f05e97a4c794e40aa63b0e9f4922480, State: Initialized, Role: FOLLOWER
I20260927 18:50:28.261247 21539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f05e97a4c794e40aa63b0e9f4922480" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41807 } }
I20260927 18:50:28.261556 21539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:28.261708 21539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:28.261921 21539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:28.264642 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:28.264894 21539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f05e97a4c794e40aa63b0e9f4922480" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41807 } }
I20260927 18:50:28.265470 21539 leader_election.cc:302] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f05e97a4c794e40aa63b0e9f4922480; no voters:
I20260927 18:50:28.266134 21539 leader_election.cc:288] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:28.266343 21544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:28.268424 21539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:28.274504 21544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [term 1 LEADER]: Becoming Leader. State: Replica: 8f05e97a4c794e40aa63b0e9f4922480, State: Running, Role: LEADER
I20260927 18:50:28.275650 21544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f05e97a4c794e40aa63b0e9f4922480" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41807 } }
I20260927 18:50:28.279637 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f05e97a4c794e40aa63b0e9f4922480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f05e97a4c794e40aa63b0e9f4922480" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41807 } } }
I20260927 18:50:28.280241 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:28.280954 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f05e97a4c794e40aa63b0e9f4922480. Latest consensus state: current_term: 1 leader_uuid: "8f05e97a4c794e40aa63b0e9f4922480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f05e97a4c794e40aa63b0e9f4922480" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41807 } } }
I20260927 18:50:28.281388 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:28.292711 21559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:28.299060 21559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:28.309152 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.277184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59317 (local address 127.31.26.254:41807)
0927 18:50:28.277434 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:28.277445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:28.287968 (+ 10523us) server_negotiation.cc:407] Connection header received
0927 18:50:28.289147 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:28.289158 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:28.289276 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:28.289408 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:28.291109 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.292441 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.294094 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.295057 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.295226 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:28.298744 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:28.298750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:28.298754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:28.298795 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:28.303088 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.304398 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.304408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.304415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.304659 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.305312 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.305322 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.305328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.305968 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:28.307931 (+ 1963us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:28.308151 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 18:50:28.308506 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:50:28.312764 21559 catalog_manager.cc:1347] Generated new cluster ID: 204f02d3a9f443cd86ea9d2cc949234c
I20260927 18:50:28.312896 21559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:28.340770 21559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:28.342319 21559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:28.352977 21559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f05e97a4c794e40aa63b0e9f4922480: Generated new TSK 0
I20260927 18:50:28.354001 21559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:28.372422 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21472
I20260927 18:50:28.373730 21534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:28.470319 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21472
2026-09-27T18:50:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2084 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:28 dist-test-slave-rfd5 krb5kdc[21571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:28 dist-test-slave-rfd5 krb5kdc[21571](info): set up 2 sockets
Sep 27 18:50:28 dist-test-slave-rfd5 krb5kdc[21571](info): commencing operation
krb5kdc: starting...
W20260927 18:50:29.577811 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:29 dist-test-slave-rfd5 krb5kdc[21571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:29.844873 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41973
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:30.209398 21584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:30.209703 21584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:30.209897 21584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:30.217464 21584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:30.217582 21584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:30.217715 21584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:30.217823 21584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:30.217890 21584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:30.228329 21584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:30.229789 21584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:30.231197 21584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:30.243563 21589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:30.243618 21590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:30.243593 21592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:30.244557 21584 server_base.cc:1034] running on GCE node
I20260927 18:50:30.245577 21584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:30.247225 21584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:30.248515 21584 hybrid_clock.cc:648] HybridClock initialized: now 1790535030248455 us; error 51 us; skew 500 ppm
I20260927 18:50:30.249023 21584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:30 dist-test-slave-rfd5 krb5kdc[21571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535030, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:30.252681 21584 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:30.254629 21584 webserver.cc:460] Webserver started at http://127.31.26.254:46081/ using document root <none> and password file <none>
I20260927 18:50:30.255429 21584 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:30.255591 21584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:30.255932 21584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:30.259244 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "b90c0a772dd84bb689c9c112f10df471"
format_stamp: "Formatted at 2026-09-27 18:50:30 on dist-test-slave-rfd5"
server_key: "75f0dc7163f620843649609b30629a75"
server_key_iv: "b795c5664737d8d99a6c62fa4e2a4043"
server_key_version: "encryptionkey@0"
I20260927 18:50:30.260092 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "b90c0a772dd84bb689c9c112f10df471"
format_stamp: "Formatted at 2026-09-27 18:50:30 on dist-test-slave-rfd5"
server_key: "75f0dc7163f620843649609b30629a75"
server_key_iv: "b795c5664737d8d99a6c62fa4e2a4043"
server_key_version: "encryptionkey@0"
I20260927 18:50:30.266126 21584 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 18:50:30.270272 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:30.272228 21584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260927 18:50:30.272467 21584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "b90c0a772dd84bb689c9c112f10df471"
format_stamp: "Formatted at 2026-09-27 18:50:30 on dist-test-slave-rfd5"
server_key: "75f0dc7163f620843649609b30629a75"
server_key_iv: "b795c5664737d8d99a6c62fa4e2a4043"
server_key_version: "encryptionkey@0"
I20260927 18:50:30.272701 21584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:30.307034 21584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:30.311403 21584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:30.311714 21584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:30.330840 21584 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41973
I20260927 18:50:30.330878 21650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41973 every 8 connection(s)
I20260927 18:50:30.332368 21584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:30.334601 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21584
I20260927 18:50:30.334822 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:30.335268 31851 external_mini_cluster.cc:1428] Setting key 5fdaf65b49dc0aae1c634ab11a48b05f
I20260927 18:50:30.336709 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:30.349943 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:30.337352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56993 (local address 127.31.26.254:41973)
0927 18:50:30.338176 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:30.338188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:30.338232 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:50:30.339851 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:30.339873 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:30.340311 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:30.340675 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:30.341335 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:30.342764 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:30.344726 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:30.345544 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:30.345688 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:30.347184 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:30.347193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:30.347210 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:30.347892 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:30.347980 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:30.348015 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:30.348143 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:30.348469 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:50:30.348828 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":370,"threads_started":1}
I20260927 18:50:30.350983 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471: Bootstrap starting.
I20260927 18:50:30.353895 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:30.354508 21651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:30.355558 21651 log.cc:824] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:30.359226 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471: No bootstrap required, opened a new log
I20260927 18:50:30.365940 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:30.358330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45451 (local address 127.31.26.254:41973)
0927 18:50:30.358487 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:30.358492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:30.359019 (+ 527us) server_negotiation.cc:407] Connection header received
0927 18:50:30.359258 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:30.359262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:30.359326 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:30.359449 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:30.360287 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:30.361021 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:30.363100 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:30.363610 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:30.363712 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:30.365004 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:30.365010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:30.365013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:30.365037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:30.365148 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:30.365180 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:30.365285 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:30.365443 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:50:30.365531 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:50:30.369544 21651 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90c0a772dd84bb689c9c112f10df471" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41973 } }
I20260927 18:50:30.370019 21651 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:30.370134 21651 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90c0a772dd84bb689c9c112f10df471, State: Initialized, Role: FOLLOWER
I20260927 18:50:30.370795 21651 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90c0a772dd84bb689c9c112f10df471" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41973 } }
I20260927 18:50:30.371047 21651 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:30.371191 21651 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:30.371379 21651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:30.374177 21651 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90c0a772dd84bb689c9c112f10df471" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41973 } }
I20260927 18:50:30.374727 21651 leader_election.cc:302] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90c0a772dd84bb689c9c112f10df471; no voters:
I20260927 18:50:30.375434 21651 leader_election.cc:288] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:30.375586 21661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:30.376828 21661 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [term 1 LEADER]: Becoming Leader. State: Replica: b90c0a772dd84bb689c9c112f10df471, State: Running, Role: LEADER
I20260927 18:50:30.377458 21661 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90c0a772dd84bb689c9c112f10df471" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41973 } }
I20260927 18:50:30.378208 21651 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:30.380579 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90c0a772dd84bb689c9c112f10df471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90c0a772dd84bb689c9c112f10df471" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41973 } } }
I20260927 18:50:30.381021 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:30.382956 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90c0a772dd84bb689c9c112f10df471. Latest consensus state: current_term: 1 leader_uuid: "b90c0a772dd84bb689c9c112f10df471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90c0a772dd84bb689c9c112f10df471" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41973 } } }
I20260927 18:50:30.383414 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:30.384032 21670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:30.389605 21670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:30.400151 21670 catalog_manager.cc:1347] Generated new cluster ID: 3cd2736baa964ef5a8b2a5fbb8befa77
I20260927 18:50:30.400291 21670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:30.414552 21670 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:30.416088 21670 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:30.425616 21670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90c0a772dd84bb689c9c112f10df471: Generated new TSK 0
I20260927 18:50:30.426563 21670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:30.432367 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21584
W20260927 18:50:30.433969 21646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56993
W20260927 18:50:30.434823 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:30.435003 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21584
2026-09-27T18:50:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:30 dist-test-slave-rfd5 krb5kdc[21682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:30 dist-test-slave-rfd5 krb5kdc[21682](info): set up 2 sockets
Sep 27 18:50:30 dist-test-slave-rfd5 krb5kdc[21682](info): commencing operation
krb5kdc: starting...
W20260927 18:50:31.545759 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:31 dist-test-slave-rfd5 krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:31.812119 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34667
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:34667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:32.168161 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:32.168558 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:32.168746 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:32.176285 21695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:32.176477 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:32.176602 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:32.176717 21695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:32.176808 21695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:32.187172 21695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:34667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:32.188645 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:32.190162 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:32.202848 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:32.202927 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:32.202878 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:32.204119 21695 server_base.cc:1034] running on GCE node
I20260927 18:50:32.204962 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:32.206516 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:32.207755 21695 hybrid_clock.cc:648] HybridClock initialized: now 1790535032207706 us; error 43 us; skew 500 ppm
I20260927 18:50:32.208192 21695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:32 dist-test-slave-rfd5 krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535032, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:32.217597 21695 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:32.219304 21695 webserver.cc:460] Webserver started at http://127.31.26.254:33897/ using document root <none> and password file <none>
I20260927 18:50:32.220038 21695 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:32.220147 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:32.220439 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:32.223613 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "55bf1ad54c7b47dca97ba3b6590657bb"
format_stamp: "Formatted at 2026-09-27 18:50:32 on dist-test-slave-rfd5"
server_key: "de3b1fb97e42bfc395cba7cf78ca8708"
server_key_iv: "ac38d1d46f0820c9e9a59bbb437f9826"
server_key_version: "encryptionkey@0"
I20260927 18:50:32.224426 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "55bf1ad54c7b47dca97ba3b6590657bb"
format_stamp: "Formatted at 2026-09-27 18:50:32 on dist-test-slave-rfd5"
server_key: "de3b1fb97e42bfc395cba7cf78ca8708"
server_key_iv: "ac38d1d46f0820c9e9a59bbb437f9826"
server_key_version: "encryptionkey@0"
I20260927 18:50:32.230290 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:50:32.234229 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:32.235764 21695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:50:32.235940 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "55bf1ad54c7b47dca97ba3b6590657bb"
format_stamp: "Formatted at 2026-09-27 18:50:32 on dist-test-slave-rfd5"
server_key: "de3b1fb97e42bfc395cba7cf78ca8708"
server_key_iv: "ac38d1d46f0820c9e9a59bbb437f9826"
server_key_version: "encryptionkey@0"
I20260927 18:50:32.236155 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:32.242486 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:32.247345 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:32.247653 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:32.267788 21695 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:34667
I20260927 18:50:32.267813 21761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:34667 every 8 connection(s)
I20260927 18:50:32.269400 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:32.273561 21762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:32.279712 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21695
I20260927 18:50:32.279891 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:32.280360 31851 external_mini_cluster.cc:1428] Setting key f4113593546895e9bfe18de552e0ad22
I20260927 18:50:32.286502 21762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb: Bootstrap starting.
I20260927 18:50:32.290042 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:32.282373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33559 (local address 127.31.26.254:34667)
0927 18:50:32.283401 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:32.283422 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:32.283475 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:50:32.285991 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:32.286023 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:32.286513 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:32.286994 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:32.287977 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:32.287985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:32.288003 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:32.288020 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:32.288092 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:32.288125 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:32.288375 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:32.288532 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 18:50:32.288752 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":457,"threads_started":1}
I20260927 18:50:32.290675 21762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:32.291980 21762 log.cc:824] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:32.294430 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:32.295408 21762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb: No bootstrap required, opened a new log
I20260927 18:50:32.301694 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:32.298947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40487 (local address 127.31.26.254:34667)
0927 18:50:32.299145 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:32.299152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:32.299589 (+ 437us) server_negotiation.cc:407] Connection header received
0927 18:50:32.299733 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:32.299739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:32.299807 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:32.299928 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:32.300911 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:32.300918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:32.300923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:32.300942 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:32.300996 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:32.301023 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:32.301199 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:32.301334 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:50:32.301398 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:50:32.304850 21762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34667 } }
I20260927 18:50:32.305459 21762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:32.305583 21762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55bf1ad54c7b47dca97ba3b6590657bb, State: Initialized, Role: FOLLOWER
I20260927 18:50:32.306183 21762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34667 } }
I20260927 18:50:32.306432 21762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:32.306586 21762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:32.306780 21762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:32.309057 21762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34667 } }
I20260927 18:50:32.309908 21762 leader_election.cc:302] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55bf1ad54c7b47dca97ba3b6590657bb; no voters:
I20260927 18:50:32.310472 21762 leader_election.cc:288] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:32.310616 21772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:32.311335 21772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [term 1 LEADER]: Becoming Leader. State: Replica: 55bf1ad54c7b47dca97ba3b6590657bb, State: Running, Role: LEADER
I20260927 18:50:32.312013 21772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34667 } }
I20260927 18:50:32.312717 21762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:32.314786 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34667 } } }
I20260927 18:50:32.315204 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:32.315872 21774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55bf1ad54c7b47dca97ba3b6590657bb. Latest consensus state: current_term: 1 leader_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf1ad54c7b47dca97ba3b6590657bb" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 34667 } } }
I20260927 18:50:32.316378 21774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:32.317183 21777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:32.322273 21777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:32.333768 21777 catalog_manager.cc:1347] Generated new cluster ID: b29bb47b99c54215b187bb1c9714f7ac
I20260927 18:50:32.333922 21777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:32.349021 21777 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:32.349510 21777 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:32.363637 21777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55bf1ad54c7b47dca97ba3b6590657bb: Generated new TSK 0
I20260927 18:50:32.364521 21777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:32.373656 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21695
W20260927 18:50:32.374888 21757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33559
W20260927 18:50:32.375628 31851 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:50:32.375797 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21695
2026-09-27T18:50:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1940 ms)
[----------] 18 tests from AuthTokenIssuingTest (35915 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:32 dist-test-slave-rfd5 krb5kdc[21793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:32 dist-test-slave-rfd5 krb5kdc[21793](info): set up 2 sockets
Sep 27 18:50:32 dist-test-slave-rfd5 krb5kdc[21793](info): commencing operation
krb5kdc: starting...
W20260927 18:50:33.489854 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:33 dist-test-slave-rfd5 krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:33.757241 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41299
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41299,127.31.26.253:37363,127.31.26.252:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:34.094998 21806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:50:34.095394 21806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:34.095527 21806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:34.095675 21806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:34.103093 21806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:34.103205 21806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:34.103318 21806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:34.103397 21806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:34.103495 21806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:34.113883 21806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41299,127.31.26.253:37363,127.31.26.252:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:34.115458 21806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:34.117362 21806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:34.130716 21811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:34.130651 21812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:34.130620 21814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:34.131737 21806 server_base.cc:1034] running on GCE node
I20260927 18:50:34.132601 21806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:34.134176 21806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:34.135401 21806 hybrid_clock.cc:648] HybridClock initialized: now 1790535034135338 us; error 37 us; skew 500 ppm
I20260927 18:50:34.135746 21806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:34 dist-test-slave-rfd5 krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:34.139053 21806 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:34.140666 21806 webserver.cc:460] Webserver started at http://127.31.26.254:33889/ using document root <none> and password file <none>
I20260927 18:50:34.141502 21806 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:34.141649 21806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:34.141976 21806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:34.145053 21806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "023299e5454e4af6b889becf20e148db"
format_stamp: "Formatted at 2026-09-27 18:50:34 on dist-test-slave-rfd5"
server_key: "1799a694a2ab83f6cc217519a0686c4b"
server_key_iv: "850a940f44b6520fa7087d028712f989"
server_key_version: "encryptionkey@0"
I20260927 18:50:34.145996 21806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "023299e5454e4af6b889becf20e148db"
format_stamp: "Formatted at 2026-09-27 18:50:34 on dist-test-slave-rfd5"
server_key: "1799a694a2ab83f6cc217519a0686c4b"
server_key_iv: "850a940f44b6520fa7087d028712f989"
server_key_version: "encryptionkey@0"
I20260927 18:50:34.151475 21806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:50:34.155238 21821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:34.156664 21806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:50:34.156872 21806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "023299e5454e4af6b889becf20e148db"
format_stamp: "Formatted at 2026-09-27 18:50:34 on dist-test-slave-rfd5"
server_key: "1799a694a2ab83f6cc217519a0686c4b"
server_key_iv: "850a940f44b6520fa7087d028712f989"
server_key_version: "encryptionkey@0"
I20260927 18:50:34.157145 21806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:34.178232 21806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:34.182978 21806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:34.183284 21806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:34.202262 21806 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:41299
I20260927 18:50:34.202312 21872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:41299 every 8 connection(s)
I20260927 18:50:34.203771 21806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:34.207609 21873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:34.213186 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21806
I20260927 18:50:34.213438 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:34.214046 31851 external_mini_cluster.cc:1428] Setting key 3db38cbe8881a9dce60b5f338a424661
I20260927 18:50:34.214370 21873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:34 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.253@KRBTEST.COM; defaulting to no policy
I20260927 18:50:34.237922 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.217745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.31.26.254:41299)
0927 18:50:34.218628 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:34.218639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:34.219227 (+ 588us) server_negotiation.cc:407] Connection header received
0927 18:50:34.221366 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:34.221372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:34.221619 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:34.221847 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:34.223185 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.224418 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.225884 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.226444 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.226651 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.230081 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:34.230090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:34.230104 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:34.230141 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:34.233056 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.233929 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.233938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.233946 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.234103 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.234502 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.234507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.234510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.235084 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:34.235299 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:34.235890 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 18:50:34.236299 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":666,"thread_start_us":460,"threads_started":1}
I20260927 18:50:34.237941 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.217767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41299 (local address 127.0.0.1:47232)
0927 18:50:34.219060 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.219100 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:34.220919 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:34.221850 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:34.221857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:34.222620 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:34.222972 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.222990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.224579 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.224588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.225745 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.225758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.226690 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.226696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.226862 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.227562 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:34.227588 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:34.229854 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:34.233250 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.233262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.233282 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.233790 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.234214 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.234221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.234224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.234390 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.235233 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:34.235241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:34.235489 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 18:50:34.235741 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 18:50:34.236429 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":196,"thread_start_us":925,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.253@KRBTEST.COM" created.
I20260927 18:50:34.241400 21873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:50:34.242678 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.242129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:43950)
0927 18:50:34.242354 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.242498 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 18:50:34.243012 21822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
I20260927 18:50:34.244822 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.244335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:43960)
0927 18:50:34.244523 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.244625 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:50:34.245060 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:37363: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab.
Entry for principal kudu/127.31.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab.
I20260927 18:50:34.272693 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:37363
--webserver_interface=127.31.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab
--principal=kudu/127.31.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41299,127.31.26.253:37363,127.31.26.252:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:34.294467 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } attempt: 1
I20260927 18:50:34.295874 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.295358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:43970)
0927 18:50:34.295570 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.295707 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 18:50:34.296955 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.296557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:43978)
0927 18:50:34.296713 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.296797 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:50:34.297173 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:37363: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
I20260927 18:50:34.365464 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } attempt: 2
I20260927 18:50:34.366852 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.366298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:43992)
0927 18:50:34.366571 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.366697 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 18:50:34.367928 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.367533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44004)
0927 18:50:34.367713 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.367788 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:50:34.368110 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:37363: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
I20260927 18:50:34.459404 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } attempt: 3
I20260927 18:50:34.460567 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.460106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44020)
0927 18:50:34.460337 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.460433 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 18:50:34.461645 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.461286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44032)
0927 18:50:34.461430 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.461506 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 18:50:34.461874 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:37363: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
I20260927 18:50:34.605191 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } attempt: 4
I20260927 18:50:34.606419 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.605868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44034)
0927 18:50:34.606181 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.606269 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 18:50:34.607383 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.607030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44050)
0927 18:50:34.607183 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.607258 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:50:34.607564 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:37363: Network error: Client connection negotiation failed: client connection to 127.31.26.253:37363: connect: Connection refused (error 111)
W20260927 18:50:34.622447 21880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:50:34.622885 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:34.623014 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:34.623163 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:34.630777 21880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:34.630930 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:34.631040 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:34.631132 21880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:34.631246 21880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:34.641850 21880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41299,127.31.26.253:37363,127.31.26.252:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab
--principal=kudu/127.31.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:34.643286 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:34.644722 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:34.657735 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:34.657703 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:34.657661 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:34.658577 21880 server_base.cc:1034] running on GCE node
I20260927 18:50:34.659471 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:34.661283 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:34.662576 21880 hybrid_clock.cc:648] HybridClock initialized: now 1790535034662501 us; error 53 us; skew 500 ppm
I20260927 18:50:34.663004 21880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:34 dist-test-slave-rfd5 krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:34.666554 21880 init.cc:377] Logged in from keytab as kudu/127.31.26.253@KRBTEST.COM (short username kudu)
I20260927 18:50:34.668190 21880 webserver.cc:460] Webserver started at http://127.31.26.253:39527/ using document root <none> and password file <none>
I20260927 18:50:34.668988 21880 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:34.669158 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:34.669513 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:34.672796 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data/instance:
uuid: "4ffb051830394150a868d5d9352ee728"
format_stamp: "Formatted at 2026-09-27 18:50:34 on dist-test-slave-rfd5"
server_key: "b3c55d8fa90b2665fa0422f269253646"
server_key_iv: "54ebeb5cdaaef58192bf26294c7c8acc"
server_key_version: "encryptionkey@0"
I20260927 18:50:34.673715 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal/instance:
uuid: "4ffb051830394150a868d5d9352ee728"
format_stamp: "Formatted at 2026-09-27 18:50:34 on dist-test-slave-rfd5"
server_key: "b3c55d8fa90b2665fa0422f269253646"
server_key_iv: "54ebeb5cdaaef58192bf26294c7c8acc"
server_key_version: "encryptionkey@0"
I20260927 18:50:34.679445 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:50:34.683146 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:34.684571 21880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:50:34.684787 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal
uuid: "4ffb051830394150a868d5d9352ee728"
format_stamp: "Formatted at 2026-09-27 18:50:34 on dist-test-slave-rfd5"
server_key: "b3c55d8fa90b2665fa0422f269253646"
server_key_iv: "54ebeb5cdaaef58192bf26294c7c8acc"
server_key_version: "encryptionkey@0"
I20260927 18:50:34.685027 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:34.711006 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:34.715910 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:34.716216 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:34.735716 21880 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.253:37363
I20260927 18:50:34.735754 21946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.253:37363 every 8 connection(s)
I20260927 18:50:34.737313 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
I20260927 18:50:34.739719 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21880
I20260927 18:50:34.739919 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-1/wal/instance
I20260927 18:50:34.740490 31851 external_mini_cluster.cc:1428] Setting key 99ef77a583210c4fd02e08d8430f1c6c
I20260927 18:50:34.742327 21947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:34.752776 21947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.252@KRBTEST.COM" created.
Sep 27 18:50:34 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:34.778680 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.755716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.31.26.254:41299)
0927 18:50:34.756300 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:34.756310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:34.757511 (+ 1201us) server_negotiation.cc:407] Connection header received
0927 18:50:34.759638 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:34.759647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:34.759750 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:34.759897 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:34.761926 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.763190 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.764943 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.765728 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.765899 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.771901 (+ 6002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:34.771908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:34.771911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:34.771968 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:34.775009 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.775960 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.775965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.775968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.776107 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.776618 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.776623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.776625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.776932 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:34.777155 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:34.777951 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 18:50:34.778197 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":270,"threads_started":1}
I20260927 18:50:34.779765 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.756399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41299 (local address 127.0.0.1:47242)
0927 18:50:34.757299 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.757349 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:34.759286 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:34.760165 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:34.760177 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:34.760921 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:34.761620 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.761641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.763376 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.763380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.764766 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.764781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.765980 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.765985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.766205 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.767015 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:34.767041 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:34.771571 (+ 4530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:34.775220 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.775232 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.775259 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.775797 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.776246 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.776252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.776256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.776468 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.777105 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:34.777116 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:34.777531 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 18:50:34.777834 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 18:50:34.778448 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":362,"threads_started":1}
I20260927 18:50:34.781829 21947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:34 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:50:34.798061 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.782616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44064)
0927 18:50:34.782832 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.782853 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:34.782973 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:34.783799 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:34.783810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:34.784355 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:34.784574 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.784589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.786103 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.786109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.787708 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.787727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.788721 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.788730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.788977 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.789715 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:34.789745 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:34.792143 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:34.795557 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.795565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.795569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.795963 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.796382 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.796390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.796395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.796546 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.797218 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:34.797221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:34.797345 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 18:50:34.797459 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:50:34.797732 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 18:50:34.798313 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.782720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.31.26.253:37363)
0927 18:50:34.783349 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:34.783355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:34.783383 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:50:34.783430 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:34.783436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:34.783629 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:34.783757 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:34.784777 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.785953 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.787907 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.788529 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.788626 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.792365 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:34.792378 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:34.792392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:34.792424 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:34.795421 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.796117 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.796124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.796126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.796256 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.796641 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.796644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.796646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.797051 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:34.797302 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:34.797637 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 18:50:34.797852 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":324,"threads_started":1}
I20260927 18:50:34.800913 21947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab.
Entry for principal kudu/127.31.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab.
I20260927 18:50:34.802290 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.801805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39250)
0927 18:50:34.801988 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.802132 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:50:34.802534 21896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:34.802765 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:40293
--webserver_interface=127.31.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab
--principal=kudu/127.31.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:41299,127.31.26.253:37363,127.31.26.252:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:34.806720 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.806272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39254)
0927 18:50:34.806469 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.806579 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 18:50:34.806948 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:34.856323 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 1
I20260927 18:50:34.857724 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.857209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39266)
0927 18:50:34.857437 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.857567 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 18:50:34.858846 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.858487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39278)
0927 18:50:34.858647 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.858725 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 18:50:34.859068 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:34.906814 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } attempt: 5
Sep 27 18:50:34 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:50:34.920583 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.907696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.31.26.253:37363)
0927 18:50:34.907869 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:34.907876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:34.907920 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:50:34.908143 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:34.908148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:34.908226 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:34.908359 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:34.909193 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.909902 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.911053 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.911529 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.911666 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.914989 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:34.914995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:34.914998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:34.915032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:34.917903 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.918601 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.918608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.918611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.918744 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:34.919126 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:34.919132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:34.919136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:34.919436 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:34.919593 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:34.920009 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 18:50:34.920229 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:50:34.920583 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.907585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44072)
0927 18:50:34.907833 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.907855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:34.907999 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:34.908387 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:34.908391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:34.908814 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:34.908995 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.909004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.910039 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.910045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.910903 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:34.910914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:34.911692 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:34.911697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:34.911927 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:34.912589 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:34.912608 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:34.914755 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:34.918063 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.918068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.918073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.918473 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.918865 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:34.918870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:34.918874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:34.919017 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:34.919558 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:34.919562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:34.919726 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 18:50:34.919909 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 18:50:34.920214 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 18:50:34.922006 21873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:50:34.923048 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.922630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39292)
0927 18:50:34.922799 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.922900 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:50:34.923925 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.923607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39298)
0927 18:50:34.923747 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.923830 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:50:34.924147 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:34.927271 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 2
I20260927 18:50:34.928257 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.927863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39314)
0927 18:50:34.928031 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.928115 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 18:50:34.929236 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.928855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39330)
0927 18:50:34.928992 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.929066 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 18:50:34.929450 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:34.975389 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 1
I20260927 18:50:34.976707 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.976105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39344)
0927 18:50:34.976367 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.976487 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 18:50:34.977828 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.977432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39354)
0927 18:50:34.977590 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.977692 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:50:34.978057 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:35.020731 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 3
I20260927 18:50:35.021996 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.021540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39366)
0927 18:50:35.021735 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.021833 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 18:50:35.023149 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.022685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39374)
0927 18:50:35.022910 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.022989 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260927 18:50:35.023401 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:35.046289 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 2
I20260927 18:50:35.047667 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.047012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39376)
0927 18:50:35.047307 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.047436 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:50:35.048820 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.048365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39388)
0927 18:50:35.048548 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.048667 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:50:35.049054 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
W20260927 18:50:35.150349 21955 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:50:35.150679 21955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:35.150751 21955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:35.150833 21955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:50:35.155336 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 3
I20260927 18:50:35.156599 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.156052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39394)
0927 18:50:35.156281 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.156389 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 18:50:35.157881 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.157466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39404)
0927 18:50:35.157655 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.157743 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 18:50:35.158147 21873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
W20260927 18:50:35.158995 21955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:35.159114 21955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:35.159219 21955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:35.159301 21955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:35.159405 21955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:35.166635 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 4
I20260927 18:50:35.167794 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.167278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39420)
0927 18:50:35.167505 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.167597 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:50:35.168817 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.168468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39426)
0927 18:50:35.168611 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.168696 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:50:35.169023 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:40293: Network error: Client connection negotiation failed: client connection to 127.31.26.252:40293: connect: Connection refused (error 111)
I20260927 18:50:35.170228 21955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:41299,127.31.26.253:37363,127.31.26.252:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab
--principal=kudu/127.31.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:35.171578 21955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:35.172937 21955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:35.185635 21961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:35.185736 21963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:35.185578 21960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:35.186451 21955 server_base.cc:1034] running on GCE node
I20260927 18:50:35.187317 21955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:35.188839 21955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:35.190096 21955 hybrid_clock.cc:648] HybridClock initialized: now 1790535035189969 us; error 88 us; skew 500 ppm
I20260927 18:50:35.190510 21955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:35.194022 21955 init.cc:377] Logged in from keytab as kudu/127.31.26.252@KRBTEST.COM (short username kudu)
I20260927 18:50:35.195722 21955 webserver.cc:460] Webserver started at http://127.31.26.252:33377/ using document root <none> and password file <none>
I20260927 18:50:35.196496 21955 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:35.196615 21955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:35.196949 21955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:35.200166 21955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data/instance:
uuid: "9974958335c74652abd051307025a918"
format_stamp: "Formatted at 2026-09-27 18:50:35 on dist-test-slave-rfd5"
server_key: "96cac2a78027e231c781d3503d9f5c37"
server_key_iv: "65031ad5b721a4ac3c50a6a3ee09779e"
server_key_version: "encryptionkey@0"
I20260927 18:50:35.201153 21955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal/instance:
uuid: "9974958335c74652abd051307025a918"
format_stamp: "Formatted at 2026-09-27 18:50:35 on dist-test-slave-rfd5"
server_key: "96cac2a78027e231c781d3503d9f5c37"
server_key_iv: "65031ad5b721a4ac3c50a6a3ee09779e"
server_key_version: "encryptionkey@0"
I20260927 18:50:35.207063 21955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:35.210907 21970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:35.212448 21955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:50:35.212626 21955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal
uuid: "9974958335c74652abd051307025a918"
format_stamp: "Formatted at 2026-09-27 18:50:35 on dist-test-slave-rfd5"
server_key: "96cac2a78027e231c781d3503d9f5c37"
server_key_iv: "65031ad5b721a4ac3c50a6a3ee09779e"
server_key_version: "encryptionkey@0"
I20260927 18:50:35.212834 21955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:35.228658 21955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:35.233651 21955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:35.233889 21955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:35.252823 21955 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.252:40293
I20260927 18:50:35.252843 22021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.252:40293 every 8 connection(s)
I20260927 18:50:35.254448 21955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
I20260927 18:50:35.258695 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:35.259654 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 21955
I20260927 18:50:35.259848 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534668899567-31851-0/minicluster-data/master-2/wal/instance
I20260927 18:50:35.260314 31851 external_mini_cluster.cc:1428] Setting key bce0e88daa0dc81bedabf97a17b5761d
I20260927 18:50:35.268172 22022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:35.285691 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.262579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.31.26.252:40293)
0927 18:50:35.263448 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.263461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.263509 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:50:35.265393 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.265422 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.265915 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.266386 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.267994 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.269470 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.270838 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.271708 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.271932 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.276872 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.276880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.276892 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.276937 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.280765 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.281807 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.281815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.281822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.282007 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.282449 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.282459 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.282463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.283058 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.283391 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.283870 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.284314 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":376,"threads_started":1}
I20260927 18:50:35.287714 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.271038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.31.26.254:41299)
0927 18:50:35.271248 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.271257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.272114 (+ 857us) server_negotiation.cc:407] Connection header received
0927 18:50:35.272400 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.272408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.272511 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.272590 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.273878 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.275020 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.276690 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.277234 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.277344 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.281545 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.281554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.281560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.281609 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.284694 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.285636 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.285641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.285643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.285797 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.286276 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.286281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.286284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.286669 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.286834 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.287281 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.287479 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:50:35.287899 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.271101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:41299 (local address 127.0.0.1:47246)
0927 18:50:35.271972 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.272024 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:35.272245 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:35.272697 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:35.272706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:35.273439 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:35.273717 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.273726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.275204 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.275213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.276509 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.276527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.277427 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.277435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.277674 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.278259 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:35.278305 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:35.281301 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:35.284827 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.284841 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.284849 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.285451 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.285957 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.285961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.285965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.286143 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.286813 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:35.286819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:35.286970 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 18:50:35.287108 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:50:35.287477 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":464,"threads_started":1}
I20260927 18:50:35.289574 22022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:35.305644 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.290396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:37363 (local address 127.0.0.1:44088)
0927 18:50:35.290579 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.290600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:35.290704 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:35.291177 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:35.291182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:35.291657 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:35.291815 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.291823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.293144 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.293150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.294533 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.294547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.295527 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.295533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.295766 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.296485 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:35.296510 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:35.299231 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:35.302842 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.302850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.302856 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.303383 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.303764 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.303769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.303773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.303965 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.304633 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:35.304637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:35.304816 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 18:50:35.304977 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 18:50:35.305332 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:50:35.305764 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.290735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.31.26.253:37363)
0927 18:50:35.290907 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.290912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.290927 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:50:35.290975 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.290979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.291037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.291122 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.291980 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.292976 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.294698 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.295366 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.295488 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.299481 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.299488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.299491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.299559 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.302675 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.303509 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.303512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.303515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.303651 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.304092 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.304096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.304099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.304493 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.304659 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.305076 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.305399 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:50:35.306953 22022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:50:35.307518 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.289401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.31.26.254:41299)
0927 18:50:35.289567 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.289573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.289647 (+ 74us) server_negotiation.cc:407] Connection header received
0927 18:50:35.289867 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.289872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.289935 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.290032 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.291650 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.292717 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.294566 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.295273 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.295441 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.300606 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.300612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.300616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.300653 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.303960 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.304860 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.304866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.304869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.305043 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.305528 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.305534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.305537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.305891 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.306228 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.306807 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.307012 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:50:35.320499 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.307713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39446)
0927 18:50:35.307877 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.307894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:35.307986 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:35.308344 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:35.308349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:35.308890 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:35.309029 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.309057 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.310173 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.310181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.311023 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.311034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.311720 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.311725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.311932 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.312534 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:35.312551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:35.314583 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:35.317820 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.317826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.317830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.318339 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.318837 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.318840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.318843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.318964 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.319667 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:35.319673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:35.319878 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 18:50:35.320007 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:50:35.320233 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:50:35.320823 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.307802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.31.26.252:40293)
0927 18:50:35.307962 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.307967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.307983 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:50:35.308080 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.308084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.308155 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.308299 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.309223 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.310053 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.311155 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.311560 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.311690 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.314780 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.314798 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.314805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.314844 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.317671 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.318487 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.318492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.318494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.318694 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.319077 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.319083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.319086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.319522 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.319674 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.320158 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.320409 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:50:35.325912 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.309572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.31.26.253:37363)
0927 18:50:35.309761 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.309767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.309786 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:50:35.309841 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.309845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.309897 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.309994 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.311184 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.311957 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.313519 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.313954 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.314084 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.318777 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.318789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.318794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.318844 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.322502 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.323539 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.323544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.323546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.323700 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.324254 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.324259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.324261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.324580 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.324834 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.325320 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.325532 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:50:35.326908 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: Bootstrap starting.
I20260927 18:50:35.331185 22022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:35.331620 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:35.332432 22022 log.cc:824] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:35.335392 21873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 4
I20260927 18:50:35.336174 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: No bootstrap required, opened a new log
I20260927 18:50:35.334956 21848 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "023299e5454e4af6b889becf20e148db"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47258
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:50:35.345942 22022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.346514 22022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9974958335c74652abd051307025a918, State: Initialized, Role: FOLLOWER
I20260927 18:50:35.347144 22022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.348474 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.348901 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.350118 22022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:35.355587 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.336324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.31.26.252:40293)
0927 18:50:35.336537 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.336543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.336592 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:50:35.336819 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.336824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.336902 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.337025 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.338144 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.339366 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.341397 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.342247 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.342393 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.347256 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.347264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.347273 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.347319 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.352092 (+ 4773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.353078 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.353132 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.353143 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.353422 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.353919 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.353925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.353928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.354349 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.354512 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.354951 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.355182 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:50:35.355575 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.336180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39454)
0927 18:50:35.336497 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.336524 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:35.336671 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:35.337109 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:35.337116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:35.337698 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:35.337940 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.337954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.339536 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.339543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.341225 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.341248 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.342417 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.342423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.342696 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.343470 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:35.343498 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:35.346982 (+ 3484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:35.352283 (+ 5301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.352289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.352293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.352889 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.353581 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.353587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.353591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.353782 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.354477 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:35.354483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:35.354654 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 18:50:35.354819 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 18:50:35.355145 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260927 18:50:35.364176 21873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db: Bootstrap starting.
I20260927 18:50:35.368835 21873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:35.370395 21873 log.cc:824] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db: Log is configured to *not* fsync() on all Append() calls
W20260927 18:50:35.374516 22039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:50:35.374673 22039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:50:35.375231 21873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db: No bootstrap required, opened a new log
I20260927 18:50:35.366263 21955 master_runner.cc:186] Error getting master registration for 127.31.26.254:41299: OK, instance_id { permanent_uuid: "023299e5454e4af6b889becf20e148db" instance_seqno: 1790535034196887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:50:35.379235 21955 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:50:35.383076 21873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.383565 21873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023299e5454e4af6b889becf20e148db, State: Initialized, Role: FOLLOWER
I20260927 18:50:35.384183 21873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.385335 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.385783 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.386766 21873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:35.386904 21848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:50:35.387137 21848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:35.390393 21848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.391851 21848 leader_election.cc:288] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [CANDIDATE]: Term 1 election: Requested vote from peers 4ffb051830394150a868d5d9352ee728 (127.31.26.253:37363), 9974958335c74652abd051307025a918 (127.31.26.252:40293)
I20260927 18:50:35.393326 21997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "023299e5454e4af6b889becf20e148db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9974958335c74652abd051307025a918"
I20260927 18:50:35.393908 21997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:35.399009 21997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023299e5454e4af6b889becf20e148db in term 1.
I20260927 18:50:35.400262 21822 leader_election.cc:302] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023299e5454e4af6b889becf20e148db, 9974958335c74652abd051307025a918; no voters:
I20260927 18:50:35.401211 22042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:35.413343 22042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [term 1 LEADER]: Becoming Leader. State: Replica: 023299e5454e4af6b889becf20e148db, State: Running, Role: LEADER
I20260927 18:50:35.415264 22042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
W20260927 18:50:35.436213 21827 tablet.cc:2369] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:35.435849 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023299e5454e4af6b889becf20e148db. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.436853 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:35.446292 22056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:35.468276 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } attempt: 5
I20260927 18:50:35.477293 22056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 18:50:35.486156 21976 tablet.cc:2369] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:50:35 dist-test-slave-rfd5 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:50:35.498396 21806 master_runner.cc:186] Error getting master registration for 127.31.26.253:37363: OK, instance_id { permanent_uuid: "4ffb051830394150a868d5d9352ee728" instance_seqno: 1790535034730069 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:50:35.511435 21806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:50:35.514976 21997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [term 1 FOLLOWER]: Refusing update from remote peer 023299e5454e4af6b889becf20e148db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:35.515971 22047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:35.526369 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.449516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.31.26.253:37363)
0927 18:50:35.449741 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.449750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.457520 (+ 7770us) server_negotiation.cc:407] Connection header received
0927 18:50:35.458537 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.458545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.458643 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.458734 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.469958 (+ 11224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.471422 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.488220 (+ 16798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.489075 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.489525 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.511654 (+ 22129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.511664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.511669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.511726 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.518689 (+ 6963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.520745 (+ 2056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.520752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.520755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.521176 (+ 421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.522689 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.522720 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.522723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.523453 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.523877 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.525121 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.525614 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":273,"server-negotiator.queue_time_us":79}
I20260927 18:50:35.531466 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.448826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.31.26.254:41299)
0927 18:50:35.449034 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.449055 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.462479 (+ 13424us) server_negotiation.cc:407] Connection header received
0927 18:50:35.463431 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.463438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.463553 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.463649 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.475104 (+ 11455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.476716 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.492343 (+ 15627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.493217 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.493369 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.512701 (+ 19332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.512713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.512718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.512768 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.520733 (+ 7965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.526021 (+ 5288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.526028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.526032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.526253 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.526824 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.526831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.526835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.527268 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.527555 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.530716 (+ 3161us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.530989 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:50:35.540817 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.469293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:40293 (local address 127.0.0.1:39468)
0927 18:50:35.469839 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.469862 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:35.469973 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:35.488345 (+ 18372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:35.488353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:35.488986 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:35.489364 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.489382 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.492216 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.492227 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.493827 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:35.493848 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.496560 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.496570 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:35.496840 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.497579 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:35.497607 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:35.500786 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:35.532282 (+ 31496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.532289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.532294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.532920 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.533543 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:35.533549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:35.533553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:35.533727 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:35.539517 (+ 5790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:35.539524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:35.539751 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 18:50:35.539916 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 18:50:35.540316 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":1}
I20260927 18:50:35.542349 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.470224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.31.26.252:40293)
0927 18:50:35.487819 (+ 17595us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.487827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.487858 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:50:35.487932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.487937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.488032 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.489748 (+ 1716us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.490079 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.491304 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.494191 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.494990 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.497255 (+ 2265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.504983 (+ 7728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.504993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.504999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.505064 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.525782 (+ 20718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.533169 (+ 7387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.533175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.533178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.533381 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.534661 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.534669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.534679 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.538368 (+ 3689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.541383 (+ 3015us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.541541 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.541824 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17421,"thread_start_us":17905,"threads_started":1}
I20260927 18:50:35.548691 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.456319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.31.26.252:40293)
0927 18:50:35.456524 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.456532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.482141 (+ 25609us) server_negotiation.cc:407] Connection header received
0927 18:50:35.483857 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.483865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.483970 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.484068 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.506464 (+ 22396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.507782 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.521812 (+ 14030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:35.522688 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:35.522819 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:35.533658 (+ 10839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:35.533665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:35.533669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:35.533716 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:35.538103 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.539495 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.539537 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.539541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.539888 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:35.543650 (+ 3762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:35.543701 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:35.543708 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:35.544539 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:35.545038 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:35.546946 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0927 18:50:35.547576 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":351104}
I20260927 18:50:35.548916 21947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: Bootstrap starting.
I20260927 18:50:35.553556 21947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:35.555105 21947 log.cc:824] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:35.568287 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023299e5454e4af6b889becf20e148db. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.568624 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.570811 21947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: No bootstrap required, opened a new log
I20260927 18:50:35.573920 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.574261 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:35.577008 22056 catalog_manager.cc:1347] Generated new cluster ID: d49569c5580c4920b6815e3dbfadd421
I20260927 18:50:35.577157 22056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:35.582594 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.582896 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.601657 22056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:35.603832 22056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:35.611900 21947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.612449 21947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffb051830394150a868d5d9352ee728, State: Initialized, Role: FOLLOWER
I20260927 18:50:35.613103 21947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } }
I20260927 18:50:35.616047 21947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:35.617213 21922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "023299e5454e4af6b889becf20e148db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ffb051830394150a868d5d9352ee728"
I20260927 18:50:35.619141 21922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:35.622735 22056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db: Generated new TSK 0
I20260927 18:50:35.624001 22056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:35.624776 21922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023299e5454e4af6b889becf20e148db in term 1.
I20260927 18:50:35.627079 21921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [term 1 FOLLOWER]: Refusing update from remote peer 023299e5454e4af6b889becf20e148db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:35.627031 22072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.627684 22072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.628630 22047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:35.667465 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.668061 22072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 023299e5454e4af6b889becf20e148db. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:35.668495 22072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.668059 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:35.688254 22089 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 18:50:35.726037 21950 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.711650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.31.26.254:41299)
0927 18:50:35.712024 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.712030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.722709 (+ 10679us) server_negotiation.cc:407] Connection header received
0927 18:50:35.724306 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.724314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.724405 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.724482 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.725759 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47276: BlockingRecv error: recv got EOF from 127.0.0.1:47276 (error 108)
Metrics: {"server-negotiator.queue_time_us":253}
W20260927 18:50:35.729640 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.711461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.31.26.253:37363)
0927 18:50:35.711877 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.711883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.727611 (+ 15728us) server_negotiation.cc:407] Connection header received
0927 18:50:35.728074 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.728081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.728256 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.728340 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.729385 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44118: BlockingRecv error: recv got EOF from 127.0.0.1:44118 (error 108)
Metrics: {"server-negotiator.queue_time_us":267}
W20260927 18:50:35.730764 22025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.723337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.31.26.252:40293)
0927 18:50:35.723528 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.723534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.727724 (+ 4190us) server_negotiation.cc:407] Connection header received
0927 18:50:35.728851 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.728859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.728972 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.729057 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.730467 (+ 1410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39478: BlockingRecv error: recv got EOF from 127.0.0.1:39478 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 18:50:35.772933 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.770604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.31.26.253:37363)
0927 18:50:35.770862 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.770871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.771429 (+ 558us) server_negotiation.cc:407] Connection header received
0927 18:50:35.771696 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.771701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.771812 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.771929 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.772705 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44134: BlockingRecv error: recv got EOF from 127.0.0.1:44134 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 18:50:35.780129 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.778029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.31.26.253:37363)
0927 18:50:35.778258 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.778267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.778777 (+ 510us) server_negotiation.cc:407] Connection header received
0927 18:50:35.778993 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.778999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.779079 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.779186 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.779907 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44146: BlockingRecv error: recv got EOF from 127.0.0.1:44146 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 18:50:35.790426 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.788205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.31.26.253:37363)
0927 18:50:35.788524 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.788532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.788803 (+ 271us) server_negotiation.cc:407] Connection header received
0927 18:50:35.789029 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.789035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.789182 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.789304 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.790161 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44152: BlockingRecv error: recv got EOF from 127.0.0.1:44152 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
W20260927 18:50:35.809643 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.807175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.31.26.253:37363)
0927 18:50:35.807489 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.807499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.808106 (+ 607us) server_negotiation.cc:407] Connection header received
0927 18:50:35.808334 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.808340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.808427 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.808547 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.809383 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44162: BlockingRecv error: recv got EOF from 127.0.0.1:44162 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
W20260927 18:50:35.823518 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.821799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.31.26.253:37363)
0927 18:50:35.821980 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.821988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.822012 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:50:35.822297 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.822302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.822380 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.822493 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.823267 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44164: BlockingRecv error: recv got EOF from 127.0.0.1:44164 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 18:50:35.846228 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.843887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.31.26.253:37363)
0927 18:50:35.844119 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.844129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.844694 (+ 565us) server_negotiation.cc:407] Connection header received
0927 18:50:35.844942 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.844946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.845035 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.845184 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.845982 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44168: BlockingRecv error: recv got EOF from 127.0.0.1:44168 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 18:50:35.884418 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.882080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.31.26.253:37363)
0927 18:50:35.882336 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.882345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.882916 (+ 571us) server_negotiation.cc:407] Connection header received
0927 18:50:35.883149 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.883153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.883237 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.883349 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.884133 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44180: BlockingRecv error: recv got EOF from 127.0.0.1:44180 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 18:50:35.954932 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:35.952436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.31.26.253:37363)
0927 18:50:35.952703 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:35.952713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:35.953356 (+ 643us) server_negotiation.cc:407] Connection header received
0927 18:50:35.953653 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:35.953658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:35.953735 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:35.953850 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:35.954685 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44184: BlockingRecv error: recv got EOF from 127.0.0.1:44184 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 18:50:36.091547 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:36.089125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.31.26.253:37363)
0927 18:50:36.089361 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.089372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.089888 (+ 516us) server_negotiation.cc:407] Connection header received
0927 18:50:36.090101 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.090105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.090193 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.090293 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:36.091276 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44200: BlockingRecv error: recv got EOF from 127.0.0.1:44200 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:50:36.126708 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "023299e5454e4af6b889becf20e148db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023299e5454e4af6b889becf20e148db" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 41299 } } peers { permanent_uuid: "4ffb051830394150a868d5d9352ee728" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 37363 } } peers { permanent_uuid: "9974958335c74652abd051307025a918" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 40293 } } }
I20260927 18:50:36.127058 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 023299e5454e4af6b889becf20e148db [sys.catalog]: This master's current role is: LEADER
W20260927 18:50:36.354787 21953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:36.352404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.31.26.253:37363)
0927 18:50:36.352691 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.352700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.353364 (+ 664us) server_negotiation.cc:407] Connection header received
0927 18:50:36.353623 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.353628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.353705 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.353792 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:36.354519 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44216: BlockingRecv error: recv got EOF from 127.0.0.1:44216 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 18:50:36.376704 22039 catalog_manager.cc:1259] Loaded cluster ID: d49569c5580c4920b6815e3dbfadd421
I20260927 18:50:36.376819 22039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: loading cluster ID for follower catalog manager: success
I20260927 18:50:36.379431 22039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: acquiring CA information for follower catalog manager: success
I20260927 18:50:36.381453 22039 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9974958335c74652abd051307025a918: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:50:36.693044 22089 catalog_manager.cc:1259] Loaded cluster ID: d49569c5580c4920b6815e3dbfadd421
I20260927 18:50:36.693202 22089 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: loading cluster ID for follower catalog manager: success
I20260927 18:50:36.696061 22089 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: acquiring CA information for follower catalog manager: success
I20260927 18:50:36.698220 22089 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ffb051830394150a868d5d9352ee728: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:50:36.880362 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.873333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.31.26.253:37363)
0927 18:50:36.874106 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.874111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.874129 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:50:36.874194 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.874198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.874341 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.874446 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:36.874988 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.875718 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.877828 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.878286 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.878388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:36.879038 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:36.879603 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:36.879717 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:36.879894 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 18:50:36.879973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":503,"threads_started":1}
I20260927 18:50:36.898008 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.891771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.31.26.252:40293)
0927 18:50:36.892524 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.892533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.892556 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:50:36.892673 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.892677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.892833 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.893021 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:36.893478 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.894235 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.895599 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.896031 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.896186 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:36.896718 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:36.897331 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:36.897506 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:36.897595 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 18:50:36.897683 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":497,"threads_started":1}
I20260927 18:50:36.920264 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.904979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.31.26.253:37363)
0927 18:50:36.905261 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.905272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.911737 (+ 6465us) server_negotiation.cc:407] Connection header received
0927 18:50:36.912008 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.912014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.912094 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.912150 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:36.914378 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.915549 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.917810 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.918454 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.918550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:36.919396 (+ 846us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:36.919532 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:36.919611 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:36.919837 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 18:50:36.919931 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 18:50:36.921265 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.905443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.31.26.254:41299)
0927 18:50:36.909515 (+ 4072us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.909523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.909545 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:50:36.909662 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.909669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.909923 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.910877 (+ 954us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:36.911037 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.912195 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.915058 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.915760 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.916533 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:36.916739 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:36.917310 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:36.920731 (+ 3421us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:36.920808 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 18:50:36.920917 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3951,"thread_start_us":3848,"threads_started":1}
I20260927 18:50:36.925518 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.912014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.31.26.252:40293)
0927 18:50:36.913535 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:36.913542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:36.918044 (+ 4502us) server_negotiation.cc:407] Connection header received
0927 18:50:36.918325 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:36.918331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:36.918428 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:36.918495 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:36.919102 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.920133 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.922072 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:36.922706 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:36.922808 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:36.924584 (+ 1776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:36.924761 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:36.924861 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:36.925126 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 18:50:36.925222 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260927 18:50:36.926762 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21806
I20260927 18:50:36.927883 21868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:37.050508 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21806
I20260927 18:50:37.072373 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21880
I20260927 18:50:37.073628 21942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:37.180327 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21880
I20260927 18:50:37.200610 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 21955
I20260927 18:50:37.201856 22017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:37.310176 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 21955
2026-09-27T18:50:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4940 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:37 dist-test-slave-rfd5 krb5kdc[22177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:37 dist-test-slave-rfd5 krb5kdc[22177](info): set up 2 sockets
Sep 27 18:50:37 dist-test-slave-rfd5 krb5kdc[22177](info): commencing operation
krb5kdc: starting...
W20260927 18:50:38.429881 31851 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/test-user.keytab.
Sep 27 18:50:38 dist-test-slave-rfd5 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.26.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
Entry for principal kudu/127.31.26.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab.
I20260927 18:50:38.704636 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44625
--webserver_interface=127.31.26.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44625,127.31.26.253:39081,127.31.26.252:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:39.050752 22190 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:50:39.051191 22190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:39.051330 22190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:39.051465 22190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:39.059275 22190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:39.059407 22190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:39.059525 22190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:39.059608 22190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:39.059711 22190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:39.070077 22190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44625,127.31.26.253:39081,127.31.26.252:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.254.keytab
--principal=kudu/127.31.26.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.254:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.26.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:39.071532 22190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:39.072958 22190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:39.085419 22195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:39.085461 22198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:39.085515 22196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:39.090919 22190 server_base.cc:1034] running on GCE node
I20260927 18:50:39.091840 22190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:39.093489 22190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:39.094763 22190 hybrid_clock.cc:648] HybridClock initialized: now 1790535039094700 us; error 42 us; skew 500 ppm
I20260927 18:50:39.095131 22190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:39 dist-test-slave-rfd5 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:39.098610 22190 init.cc:377] Logged in from keytab as kudu/127.31.26.254@KRBTEST.COM (short username kudu)
I20260927 18:50:39.100320 22190 webserver.cc:460] Webserver started at http://127.31.26.254:34077/ using document root <none> and password file <none>
I20260927 18:50:39.101145 22190 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:39.101279 22190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:39.101615 22190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:39.104743 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data/instance:
uuid: "41be32fd29284753b858a66ec1b417cf"
format_stamp: "Formatted at 2026-09-27 18:50:39 on dist-test-slave-rfd5"
server_key: "95fb12eca212bdad69c88298d70b7c49"
server_key_iv: "1bd729253d85a3c5915e17dacc53115c"
server_key_version: "encryptionkey@0"
I20260927 18:50:39.105628 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal/instance:
uuid: "41be32fd29284753b858a66ec1b417cf"
format_stamp: "Formatted at 2026-09-27 18:50:39 on dist-test-slave-rfd5"
server_key: "95fb12eca212bdad69c88298d70b7c49"
server_key_iv: "1bd729253d85a3c5915e17dacc53115c"
server_key_version: "encryptionkey@0"
I20260927 18:50:39.111284 22190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:39.115020 22205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:39.116540 22190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:50:39.116744 22190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal
uuid: "41be32fd29284753b858a66ec1b417cf"
format_stamp: "Formatted at 2026-09-27 18:50:39 on dist-test-slave-rfd5"
server_key: "95fb12eca212bdad69c88298d70b7c49"
server_key_iv: "1bd729253d85a3c5915e17dacc53115c"
server_key_version: "encryptionkey@0"
I20260927 18:50:39.116979 22190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:39.138895 22190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:39.143808 22190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:39.144104 22190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:39.163347 22190 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.254:44625
I20260927 18:50:39.163372 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.254:44625 every 8 connection(s)
I20260927 18:50:39.164834 22190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:39.168831 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:39.171319 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 22190
I20260927 18:50:39.171525 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-0/wal/instance
I20260927 18:50:39.172075 31851 external_mini_cluster.cc:1428] Setting key bfd138c68838978743e2a8b2fd215663
I20260927 18:50:39.179013 22257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:39 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.26.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.253@KRBTEST.COM" created.
I20260927 18:50:39.204057 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.183537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.31.26.254:44625)
0927 18:50:39.184547 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:39.184565 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:39.184786 (+ 221us) server_negotiation.cc:407] Connection header received
0927 18:50:39.186966 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:39.186972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:39.187232 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:39.187440 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:39.188780 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.190052 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.191500 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.192074 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.192285 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.196147 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:39.196156 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:39.196169 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:39.196198 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:39.199216 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.200014 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.200021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.200023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.200236 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.200626 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.200629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.200631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.201333 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:39.201555 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:39.202091 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 18:50:39.202508 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":793,"spinlock_wait_cycles":16128,"thread_start_us":507,"threads_started":1}
I20260927 18:50:39.204077 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.183537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44625 (local address 127.0.0.1:38792)
0927 18:50:39.184620 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.184668 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:39.186563 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:39.187458 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:39.187466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:39.188214 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:39.188566 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.188584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.190251 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.190259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.191358 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.191372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.192324 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.192331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.192504 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.193315 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:39.193340 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:39.195913 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:39.199460 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.199472 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.199482 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.199890 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.200375 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.200379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.200381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.200531 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.201472 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:39.201480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:39.201705 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 18:50:39.201966 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 18:50:39.202638 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":174,"thread_start_us":424,"threads_started":1}
I20260927 18:50:39.207450 22257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:50:39.208572 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.208159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58390)
0927 18:50:39.208341 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.208452 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:50:39.208949 22209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
I20260927 18:50:39.210763 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.210255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58400)
0927 18:50:39.210465 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.210565 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 18:50:39.211007 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:39081: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab.
Entry for principal kudu/127.31.26.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab.
I20260927 18:50:39.232529 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:39081
--webserver_interface=127.31.26.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab
--principal=kudu/127.31.26.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44625,127.31.26.253:39081,127.31.26.252:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:39.260454 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } attempt: 1
I20260927 18:50:39.261888 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.261264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58414)
0927 18:50:39.261503 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.261672 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:50:39.263093 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.262615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58426)
0927 18:50:39.262777 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.262880 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:50:39.263329 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:39081: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
I20260927 18:50:39.331636 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } attempt: 2
I20260927 18:50:39.332898 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.332375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58430)
0927 18:50:39.332630 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.332733 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 18:50:39.334219 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.333757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58436)
0927 18:50:39.333943 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.334048 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 18:50:39.334491 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:39081: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
I20260927 18:50:39.425791 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } attempt: 3
I20260927 18:50:39.427016 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.426541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58446)
0927 18:50:39.426773 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.426869 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:50:39.428164 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.427780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58456)
0927 18:50:39.427943 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.428017 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:50:39.428421 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:39081: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
I20260927 18:50:39.571730 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } attempt: 4
I20260927 18:50:39.573208 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.572530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58472)
0927 18:50:39.572908 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.572998 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260927 18:50:39.574448 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.574055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58484)
0927 18:50:39.574217 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.574315 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:50:39.574630 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.253:39081: Network error: Client connection negotiation failed: client connection to 127.31.26.253:39081: connect: Connection refused (error 111)
W20260927 18:50:39.580715 22264 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:50:39.581169 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:39.581301 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:39.581446 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:39.589175 22264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:39.589319 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:39.589416 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:39.589505 22264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:39.589596 22264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:39.600078 22264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44625,127.31.26.253:39081,127.31.26.252:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.253.keytab
--principal=kudu/127.31.26.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.253:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.26.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:39.601614 22264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:39.602988 22264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:39.615753 22269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:39.615723 22270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:39.616005 22272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:39.616712 22264 server_base.cc:1034] running on GCE node
I20260927 18:50:39.617681 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:39.619272 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:39.620568 22264 hybrid_clock.cc:648] HybridClock initialized: now 1790535039620476 us; error 48 us; skew 500 ppm
I20260927 18:50:39.620952 22264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:39 dist-test-slave-rfd5 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:39.624487 22264 init.cc:377] Logged in from keytab as kudu/127.31.26.253@KRBTEST.COM (short username kudu)
I20260927 18:50:39.626216 22264 webserver.cc:460] Webserver started at http://127.31.26.253:36989/ using document root <none> and password file <none>
I20260927 18:50:39.627020 22264 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:39.627177 22264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:39.627522 22264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:39.630882 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data/instance:
uuid: "4fbc0bef76ee4c53b241f35db78c3088"
format_stamp: "Formatted at 2026-09-27 18:50:39 on dist-test-slave-rfd5"
server_key: "8998ac5c7195f57d97e3ad991d6c66a3"
server_key_iv: "bbee8035c6c3bddaf26d012f96b76cc3"
server_key_version: "encryptionkey@0"
I20260927 18:50:39.631791 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal/instance:
uuid: "4fbc0bef76ee4c53b241f35db78c3088"
format_stamp: "Formatted at 2026-09-27 18:50:39 on dist-test-slave-rfd5"
server_key: "8998ac5c7195f57d97e3ad991d6c66a3"
server_key_iv: "bbee8035c6c3bddaf26d012f96b76cc3"
server_key_version: "encryptionkey@0"
I20260927 18:50:39.637526 22264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 18:50:39.641350 22279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:39.642972 22264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:39.643198 22264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal
uuid: "4fbc0bef76ee4c53b241f35db78c3088"
format_stamp: "Formatted at 2026-09-27 18:50:39 on dist-test-slave-rfd5"
server_key: "8998ac5c7195f57d97e3ad991d6c66a3"
server_key_iv: "bbee8035c6c3bddaf26d012f96b76cc3"
server_key_version: "encryptionkey@0"
I20260927 18:50:39.643453 22264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:39.661764 22264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:39.666447 22264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:39.666766 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:39.686914 22264 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.253:39081
I20260927 18:50:39.686935 22330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.253:39081 every 8 connection(s)
I20260927 18:50:39.688540 22264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/data/info.pb
I20260927 18:50:39.689584 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 22264
I20260927 18:50:39.689836 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-1/wal/instance
I20260927 18:50:39.690305 31851 external_mini_cluster.cc:1428] Setting key a3b286765bbfdf57bdc987b337464c89
I20260927 18:50:39.692844 22331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:39.701502 22331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.26.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.26.252@KRBTEST.COM" created.
Sep 27 18:50:39 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:39.726138 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.705418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.31.26.254:44625)
0927 18:50:39.705980 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:39.705989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:39.707077 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 18:50:39.709136 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:39.709146 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:39.709250 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:39.709392 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:39.711277 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.712578 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.714377 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.715136 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.715298 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.719471 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:39.719478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:39.719481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:39.719542 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:39.722620 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.723528 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.723533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.723535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.723669 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.724152 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.724156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.724158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.724462 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:39.724622 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:39.725502 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 18:50:39.725744 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260927 18:50:39.727245 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.706031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44625 (local address 127.0.0.1:38796)
0927 18:50:39.706878 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.706928 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:39.708806 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:39.709676 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:39.709686 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:39.710312 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:39.710975 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.711005 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.712794 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.712799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.714194 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.714210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.715433 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.715442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.715689 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.716371 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:39.716396 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:39.719186 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:39.722817 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.722828 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.722854 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.723374 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.723788 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.723801 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.723805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.724013 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.724600 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:39.724609 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:39.725041 (+ 432us) client_negotiation.cc:769] Sending connection context
0927 18:50:39.725394 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 18:50:39.725995 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":367,"threads_started":1}
I20260927 18:50:39.729302 22331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:39 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:50:39.745713 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.730371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58492)
0927 18:50:39.730573 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.730601 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:39.730717 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:39.731697 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:39.731701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:39.732489 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:39.732650 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.732660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.734240 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.734248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.735096 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.735108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.736008 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.736013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.736188 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.736883 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:39.736900 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:39.739358 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:39.742953 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.742961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.742965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.743411 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.743800 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.743807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.743813 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.743952 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.744672 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:39.744676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:39.744818 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:50:39.744960 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 18:50:39.745304 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 18:50:39.745854 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.730541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.31.26.253:39081)
0927 18:50:39.731250 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:39.731256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:39.731281 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:50:39.731331 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:39.731336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:39.731529 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:39.731706 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:39.732858 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.734082 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.735238 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.735832 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.735970 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.739581 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:39.739596 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:39.739620 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:39.739651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:39.742754 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.743549 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.743563 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.743566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.743687 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.744066 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.744075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.744077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.744551 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:39.744779 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:39.745211 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 18:50:39.745466 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":420,"threads_started":1}
I20260927 18:50:39.748409 22331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.26.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab.
Entry for principal kudu/127.31.26.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab.
I20260927 18:50:39.749681 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.749154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42764)
0927 18:50:39.749368 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.749517 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:50:39.750028 22283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:39.750651 31851 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
/tmp/dist-test-taskk9LG2U/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:46457
--webserver_interface=127.31.26.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab
--principal=kudu/127.31.26.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.26.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.26.254:44625,127.31.26.253:39081,127.31.26.252:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:39.751904 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.751433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42766)
0927 18:50:39.751619 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.751744 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 18:50:39.752210 22331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:39.801587 22331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 1
I20260927 18:50:39.803164 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.802432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42780)
0927 18:50:39.802716 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.802924 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 18:50:39.804529 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.803954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42790)
0927 18:50:39.804204 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.804339 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:50:39.804849 22331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:39.873162 22331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 2
I20260927 18:50:39.873874 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } attempt: 5
I20260927 18:50:39.874444 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.873884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42802)
0927 18:50:39.874097 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.874212 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:50:39.875667 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.875317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42816)
0927 18:50:39.875441 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.875521 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 18:50:39.875979 22331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Sep 27 18:50:39 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:50:39.887934 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.874595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.31.26.253:39081)
0927 18:50:39.874756 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:39.874762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:39.874925 (+ 163us) server_negotiation.cc:407] Connection header received
0927 18:50:39.875083 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:39.875088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:39.875180 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:39.875291 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:39.876174 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.876879 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.878113 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.878561 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.878703 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.882346 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:39.882352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:39.882356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:39.882393 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:39.885142 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.885885 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.885892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.885895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.886033 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:39.886426 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:39.886431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:39.886434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:39.886736 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:39.886915 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:39.887332 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 18:50:39.887581 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:50:39.887941 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.874507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58502)
0927 18:50:39.874765 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.874783 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:39.874940 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:39.875321 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:39.875326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:39.875767 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:39.876024 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.876038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.877018 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.877023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.877960 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:39.877973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:39.878720 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:39.878726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:39.878984 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:39.879711 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:39.879729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:39.882093 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:39.885331 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.885338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.885342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.885747 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.886153 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:39.886159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:39.886162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:39.886315 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:39.886870 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:39.886880 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:39.887060 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 18:50:39.887216 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 18:50:39.887582 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260927 18:50:39.889163 22257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:50:39.890368 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.889932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42820)
0927 18:50:39.890104 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.890186 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:50:39.891395 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.891050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42824)
0927 18:50:39.891179 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.891265 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:50:39.891710 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:39.942972 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 1
I20260927 18:50:39.944532 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.943844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42838)
0927 18:50:39.944117 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.944265 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 18:50:39.946137 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.945636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42852)
0927 18:50:39.945846 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.945953 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 18:50:39.946484 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:39.967242 22331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 3
I20260927 18:50:39.968695 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.967963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42868)
0927 18:50:39.968276 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.968439 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 18:50:39.970304 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.969647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42872)
0927 18:50:39.969946 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.970079 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260927 18:50:39.970624 22331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:40.014736 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 2
I20260927 18:50:40.015920 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.015422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42876)
0927 18:50:40.015663 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.015759 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:50:40.016976 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.016616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42878)
0927 18:50:40.016769 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.016853 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:50:40.017252 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
I20260927 18:50:40.113870 22331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 4
I20260927 18:50:40.115181 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.114566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42892)
0927 18:50:40.114900 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.114997 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 18:50:40.116374 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.115988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42898)
0927 18:50:40.116164 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.116237 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 18:50:40.116601 22331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
W20260927 18:50:40.117560 22339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:50:40.118091 22339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:40.118213 22339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:40.118358 22339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:50:40.123482 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 3
I20260927 18:50:40.124960 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.124240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42902)
0927 18:50:40.124509 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.124674 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 18:50:40.126297 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.125882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42912)
0927 18:50:40.126073 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.126168 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:50:40.126596 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.26.252:46457: Network error: Client connection negotiation failed: client connection to 127.31.26.252:46457: connect: Connection refused (error 111)
W20260927 18:50:40.130121 22339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:40.130257 22339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:40.130357 22339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:40.130445 22339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:40.130542 22339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:40.141253 22339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.26.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.26.254:44625,127.31.26.253:39081,127.31.26.252:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/krb5kdc/kudu_127.31.26.252.keytab
--principal=kudu/127.31.26.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.26.252:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.26.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:50:40.142648 22339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:40.144055 22339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:40.157240 22344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:40.157282 22345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:40.157378 22347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:40.159358 22339 server_base.cc:1034] running on GCE node
I20260927 18:50:40.160284 22339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:40.162020 22339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:40.163291 22339 hybrid_clock.cc:648] HybridClock initialized: now 1790535040163214 us; error 56 us; skew 500 ppm
I20260927 18:50:40.163654 22339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535040, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:40.167276 22339 init.cc:377] Logged in from keytab as kudu/127.31.26.252@KRBTEST.COM (short username kudu)
I20260927 18:50:40.170478 22339 webserver.cc:460] Webserver started at http://127.31.26.252:34445/ using document root <none> and password file <none>
I20260927 18:50:40.171275 22339 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:40.171411 22339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:40.171748 22339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:40.175048 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data/instance:
uuid: "b07bef3922714c4ab8aa659c7475d634"
format_stamp: "Formatted at 2026-09-27 18:50:40 on dist-test-slave-rfd5"
server_key: "59b108c9704c68dee0414b4791cb658c"
server_key_iv: "d6a1676e3e5e99daeda02de0020a3a00"
server_key_version: "encryptionkey@0"
I20260927 18:50:40.175933 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal/instance:
uuid: "b07bef3922714c4ab8aa659c7475d634"
format_stamp: "Formatted at 2026-09-27 18:50:40 on dist-test-slave-rfd5"
server_key: "59b108c9704c68dee0414b4791cb658c"
server_key_iv: "d6a1676e3e5e99daeda02de0020a3a00"
server_key_version: "encryptionkey@0"
I20260927 18:50:40.181746 22339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:50:40.185829 22354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:40.187318 22339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:40.187536 22339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data,/tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal
uuid: "b07bef3922714c4ab8aa659c7475d634"
format_stamp: "Formatted at 2026-09-27 18:50:40 on dist-test-slave-rfd5"
server_key: "59b108c9704c68dee0414b4791cb658c"
server_key_iv: "d6a1676e3e5e99daeda02de0020a3a00"
server_key_version: "encryptionkey@0"
I20260927 18:50:40.187781 22339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:40.206449 22339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:40.211059 22339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:40.211364 22339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:40.230898 22339 rpc_server.cc:307] RPC server started. Bound to: 127.31.26.252:46457
I20260927 18:50:40.230919 22405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.26.252:46457 every 8 connection(s)
I20260927 18:50:40.232389 22339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/data/info.pb
I20260927 18:50:40.236371 22406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:40.237986 31851 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu as pid 22339
I20260927 18:50:40.238180 31851 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk9LG2U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534668899567-31851-0/minicluster-data/master-2/wal/instance
I20260927 18:50:40.238726 31851 external_mini_cluster.cc:1428] Setting key 739b22e35a6642f4ca6b616dbbe14fa6
I20260927 18:50:40.246196 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535040, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:40.263945 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.240693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.31.26.252:46457)
0927 18:50:40.241617 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.241630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.241676 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:50:40.243422 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.243448 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.243923 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.244370 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.246054 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.247553 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.249112 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.250075 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.250336 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.254713 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.254722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.254736 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.254787 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.258480 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.259494 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.259502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.259510 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.259701 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.260186 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.260192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.260197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.261086 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.261437 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.261905 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.262404 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":458,"threads_started":1}
I20260927 18:50:40.266652 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.248958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.254:44625 (local address 127.0.0.1:38798)
0927 18:50:40.250043 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.250087 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:40.250257 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:40.250874 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:40.250881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:40.251569 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:40.251855 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.251864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.253036 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.253043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.254247 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.254262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.255092 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.255100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.255304 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.255983 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:40.256016 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:40.258663 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:40.262797 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.262806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.262812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.263256 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.263736 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.263740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.263743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.263905 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.265627 (+ 1722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:40.265634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:40.265820 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 18:50:40.265988 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 18:50:40.266253 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":619,"threads_started":1}
I20260927 18:50:40.266741 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.248958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.31.26.254:44625)
0927 18:50:40.249588 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.249594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.250374 (+ 780us) server_negotiation.cc:407] Connection header received
0927 18:50:40.250556 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.250561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.250656 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.250805 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.252027 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.252818 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.254432 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.254880 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.255049 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.258911 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.258919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.258923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.258973 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.262642 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.263405 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.263411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.263415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.263583 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.264053 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.264058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.264061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.265454 (+ 1393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.265640 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.266137 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.266398 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":338,"threads_started":1}
I20260927 18:50:40.268229 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535040, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.254@KRBTEST.COM
I20260927 18:50:40.282836 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.269115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.253:39081 (local address 127.0.0.1:58508)
0927 18:50:40.269379 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.269394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:40.269533 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:40.269858 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:40.269863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:40.270287 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:40.270437 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.270448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.271416 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.271423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.272315 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.272326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.273052 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.273059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.273293 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.273847 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:40.273865 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:40.276210 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:40.280167 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.280172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.280175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.280592 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.281189 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.281195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.281199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.281381 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.281972 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:40.281978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:40.282190 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 18:50:40.282302 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:50:40.282526 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260927 18:50:40.283362 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.269054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.31.26.253:39081)
0927 18:50:40.269264 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.269269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.269471 (+ 202us) server_negotiation.cc:407] Connection header received
0927 18:50:40.269655 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.269661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.269734 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.269836 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.270590 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.271277 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.272454 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.272891 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.273021 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.276458 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.276464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.276467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.276501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.279983 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.280844 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.280849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.280851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.280993 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.281508 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.281513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.281517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.281845 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.281982 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.282479 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.282750 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 18:50:40.283824 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.267551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.31.26.254:44625)
0927 18:50:40.267719 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.267724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.267829 (+ 105us) server_negotiation.cc:407] Connection header received
0927 18:50:40.268014 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.268019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.268102 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.268246 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.269801 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.270516 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.271762 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.272202 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.272344 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.277177 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.277187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.277192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.277239 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.280543 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.281578 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.281583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.281585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.281735 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.282220 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.282224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.282227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.282612 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.283049 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.283245 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.283499 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:50:40.284396 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535040, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.252@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.26.253@KRBTEST.COM
I20260927 18:50:40.299345 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.285223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.31.26.252:46457)
0927 18:50:40.285413 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.285419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.285467 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:50:40.285675 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.285680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.285753 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.285816 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.286798 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.287477 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.288583 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.288987 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.289152 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.292819 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.292826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.292834 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.292879 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.296373 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.297205 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.297212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.297216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.297363 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.297799 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.297803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.297805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.298089 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.298240 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.298711 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.299064 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":63}
I20260927 18:50:40.299388 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.285224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42926)
0927 18:50:40.285396 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.285416 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:40.285534 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:40.285988 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:40.285992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:40.286508 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:40.286672 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.286681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.287609 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.287615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.288453 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.288463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.289212 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.289219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.289412 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.290072 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:40.290093 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:40.292608 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:40.296513 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.296521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.296527 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.297052 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.297489 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.297496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.297501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.297668 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.298208 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:40.298213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:40.298372 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 18:50:40.298572 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 18:50:40.298927 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:50:40.303135 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.285885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.31.26.253:39081)
0927 18:50:40.286070 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.286076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.286092 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:50:40.286151 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.286158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.286225 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.286318 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.287590 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.288306 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.289576 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.290175 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.290321 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.296387 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.296393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.296396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.296457 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.300073 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.301028 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.301037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.301040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.301274 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.301673 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.301679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.301683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.301973 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.302265 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.302564 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.302816 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:50:40.303844 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 4
I20260927 18:50:40.304842 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: Bootstrap starting.
I20260927 18:50:40.308959 31851 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:40.309115 22406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:40.310310 22406 log.cc:824] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:40.313550 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: No bootstrap required, opened a new log
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.254@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:50:40.312249 22232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "41be32fd29284753b858a66ec1b417cf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38812
I20260927 18:50:40.321774 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.304526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:42938)
0927 18:50:40.304733 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.304755 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:40.304874 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:40.305992 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:40.305998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:40.306460 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:40.306667 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.306677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.307865 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.307872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.309192 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.309208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.310251 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.310258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.310486 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.311218 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:40.311243 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:40.314459 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:40.318690 (+ 4231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.318696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.318700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.319132 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.319581 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.319587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.319590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.319824 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.320618 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:40.320624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:40.320849 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 18:50:40.320998 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:50:40.321348 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:50:40.321774 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.304929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.31.26.252:46457)
0927 18:50:40.305187 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.305193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.305219 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:50:40.305267 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.305270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.305329 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.305432 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.306823 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.307701 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.309376 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.310067 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.312356 (+ 2289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.314722 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.314728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.314733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.314773 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.318492 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.319278 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.319285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.319289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.319448 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.319970 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.319979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.319983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.320476 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.320657 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.321136 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.321384 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:50:40.323637 22406 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.324033 22406 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07bef3922714c4ab8aa659c7475d634, State: Initialized, Role: FOLLOWER
I20260927 18:50:40.324532 22406 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.325529 22413 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.325934 22413 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:40.326937 22406 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:40.327682 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf: Bootstrap starting.
I20260927 18:50:40.331928 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:40.333369 22257 log.cc:824] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:40.336967 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf: No bootstrap required, opened a new log
W20260927 18:50:40.340201 22425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:50:40.341506 22425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:50:40.342823 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.343300 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41be32fd29284753b858a66ec1b417cf, State: Initialized, Role: FOLLOWER
I20260927 18:50:40.343864 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.344957 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.345379 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:40.346413 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:40.349318 22232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:50:40.349529 22232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:40.353055 22232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.356863 22381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41be32fd29284753b858a66ec1b417cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b07bef3922714c4ab8aa659c7475d634"
I20260927 18:50:40.357447 22381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:40.358351 22232 leader_election.cc:288] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [CANDIDATE]: Term 1 election: Requested vote from peers 4fbc0bef76ee4c53b241f35db78c3088 (127.31.26.253:39081), b07bef3922714c4ab8aa659c7475d634 (127.31.26.252:46457)
I20260927 18:50:40.361608 22381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41be32fd29284753b858a66ec1b417cf in term 1.
I20260927 18:50:40.362692 22209 leader_election.cc:302] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41be32fd29284753b858a66ec1b417cf, b07bef3922714c4ab8aa659c7475d634; no voters:
I20260927 18:50:40.363361 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:40.343824 22339 master_runner.cc:186] Error getting master registration for 127.31.26.254:44625: OK, instance_id { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" instance_seqno: 1790535039157786 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:50:40.364202 22339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:50:40.369724 22427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [term 1 LEADER]: Becoming Leader. State: Replica: 41be32fd29284753b858a66ec1b417cf, State: Running, Role: LEADER
I20260927 18:50:40.370420 22427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.378351 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41be32fd29284753b858a66ec1b417cf. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.378988 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:40.380784 22439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 18:50:40.397138 22211 tablet.cc:2369] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:40.403525 22439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:40.415783 22331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } attempt: 5
I20260927 18:50:40.433298 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.394231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.31.26.254:44625)
0927 18:50:40.394428 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.394438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.403141 (+ 8703us) server_negotiation.cc:407] Connection header received
0927 18:50:40.405153 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.405159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.405253 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.405345 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.412711 (+ 7366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.413947 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.417820 (+ 3873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.418627 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.418781 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.424483 (+ 5702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.424492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.424498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.424548 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.429037 (+ 4489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.430586 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.430593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.430597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.430772 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.431220 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.431225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.431229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.431621 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.431906 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.432521 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.432821 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:50:40.422086 22190 master_runner.cc:186] Error getting master registration for 127.31.26.253:39081: OK, instance_id { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" instance_seqno: 1790535039681021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:50:40.437270 22190 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:50:40.442620 22381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [term 1 FOLLOWER]: Refusing update from remote peer 41be32fd29284753b858a66ec1b417cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:40.443701 22433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 18:50:40 dist-test-slave-rfd5 krb5kdc[22177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.26.253@KRBTEST.COM for kudu/127.31.26.252@KRBTEST.COM
I20260927 18:50:40.458197 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.398100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.31.26.253:39081)
0927 18:50:40.398297 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.398304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.409688 (+ 11384us) server_negotiation.cc:407] Connection header received
0927 18:50:40.410208 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.410218 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.410325 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.410399 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.416169 (+ 5770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.417263 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.424344 (+ 7081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.425073 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.425288 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.438592 (+ 13304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.438600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.438605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.438654 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.442548 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.451806 (+ 9258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.451813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.451816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.452017 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.453372 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.453378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.453381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.453765 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.454017 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.457531 (+ 3514us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.457818 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:50:40.461283 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.403844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.31.26.252:46457)
0927 18:50:40.404052 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.404061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.413584 (+ 9523us) server_negotiation.cc:407] Connection header received
0927 18:50:40.413829 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.413838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.413927 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.414003 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.422392 (+ 8389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.423551 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.426220 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.426976 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.427132 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.444930 (+ 17798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.444939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.444952 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.444998 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.450152 (+ 5154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.454559 (+ 4407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.454567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.454570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.454945 (+ 375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.456182 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.456195 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.456199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.456852 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.457276 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.460111 (+ 2835us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.460558 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 18:50:40.463634 22360 tablet.cc:2369] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:40.464396 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.416453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.26.252:46457 (local address 127.0.0.1:59436)
0927 18:50:40.416828 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.416847 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:40.416949 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:40.441573 (+ 24624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:40.441580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:40.442796 (+ 1216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:40.443017 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.443030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.446128 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.446136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.448600 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:40.448622 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.450329 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.450336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:40.450585 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.451357 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:40.451383 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:40.454460 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:40.460034 (+ 5574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.460039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.460043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.460700 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.461822 (+ 1122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:40.461828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:40.461833 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:40.462087 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:40.462888 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:40.462896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:40.463097 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 18:50:40.463676 (+ 579us) client_negotiation.cc:240] Negotiation successful
0927 18:50:40.464012 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260927 18:50:40.465802 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.417194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.31.26.252:46457)
0927 18:50:40.437457 (+ 20263us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.437465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.437492 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:50:40.437561 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.437568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.437659 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.442051 (+ 4392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.444513 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.445940 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.448830 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:40.450088 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:40.451109 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:40.454746 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:40.454753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:40.454759 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:40.454823 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:40.459854 (+ 5031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.461063 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.461071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.461078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.461333 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:40.462264 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:40.462272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:40.462278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:40.462701 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:40.464774 (+ 2073us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:40.464926 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 18:50:40.465215 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20090,"thread_start_us":20558,"threads_started":1}
I20260927 18:50:40.471705 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: Bootstrap starting.
I20260927 18:50:40.475947 22331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:40.476778 22413 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41be32fd29284753b858a66ec1b417cf. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.477145 22413 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:40.480343 22450 mvcc.cc:205] Tried to move back new op lower bound from 7334031525641580544 to 7334031525363318784. Current Snapshot: MvccSnapshot[applied={T|T < 7334031525641580544}]
I20260927 18:50:40.480790 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.481274 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:40.481640 22452 mvcc.cc:205] Tried to move back new op lower bound from 7334031525641580544 to 7334031525363318784. Current Snapshot: MvccSnapshot[applied={T|T < 7334031525641580544}]
I20260927 18:50:40.485365 22439 catalog_manager.cc:1347] Generated new cluster ID: 64e8d4c813414210843fcf8a6c103d27
I20260927 18:50:40.485479 22439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:40.489440 22331 log.cc:824] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:40.496387 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: No bootstrap required, opened a new log
I20260927 18:50:40.508556 22331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.509702 22413 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.509989 22413 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:40.513162 22331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbc0bef76ee4c53b241f35db78c3088, State: Initialized, Role: FOLLOWER
I20260927 18:50:40.517053 22439 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:40.514434 22331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } }
I20260927 18:50:40.519286 22439 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:40.529742 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.530260 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:40.533499 22305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:40.536594 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41be32fd29284753b858a66ec1b417cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fbc0bef76ee4c53b241f35db78c3088"
I20260927 18:50:40.542758 22305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [term 1 FOLLOWER]: Refusing update from remote peer 41be32fd29284753b858a66ec1b417cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:40.543519 22439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf: Generated new TSK 0
I20260927 18:50:40.544669 22439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:40.546705 22306 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41be32fd29284753b858a66ec1b417cf in term 1.
I20260927 18:50:40.552862 22433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:40.567776 22331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:40.606153 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41be32fd29284753b858a66ec1b417cf. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.606527 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:50:40.607045 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.607586 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:50:40.675606 22410 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.668184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.31.26.254:44625)
0927 18:50:40.668437 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.668444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.672383 (+ 3939us) server_negotiation.cc:407] Connection header received
0927 18:50:40.673383 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.673393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.673519 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.673622 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.675241 (+ 1619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58206: BlockingRecv error: recv got EOF from 127.0.0.1:58206 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:50:40.679920 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.672356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.31.26.253:39081)
0927 18:50:40.672576 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.672585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.677671 (+ 5086us) server_negotiation.cc:407] Connection header received
0927 18:50:40.677857 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.677865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.678004 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.678123 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.679591 (+ 1468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42672: BlockingRecv error: recv got EOF from 127.0.0.1:42672 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:50:40.681313 22480 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 18:50:40.684087 22451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.681126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.31.26.252:46457)
0927 18:50:40.681329 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.681340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.681758 (+ 418us) server_negotiation.cc:407] Connection header received
0927 18:50:40.682454 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.682459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.682574 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.682658 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.683843 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59452: BlockingRecv error: recv got EOF from 127.0.0.1:59452 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:50:40.717816 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.698908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.31.26.253:39081)
0927 18:50:40.699091 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.699098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.715794 (+ 16696us) server_negotiation.cc:407] Connection header received
0927 18:50:40.716056 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.716063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.716202 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.716334 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.717506 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42682: BlockingRecv error: recv got EOF from 127.0.0.1:42682 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 18:50:40.739250 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.737027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.31.26.253:39081)
0927 18:50:40.737311 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.737321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.737852 (+ 531us) server_negotiation.cc:407] Connection header received
0927 18:50:40.738082 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.738087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.738162 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.738272 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.739027 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42688: BlockingRecv error: recv got EOF from 127.0.0.1:42688 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260927 18:50:40.750561 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.748406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.31.26.253:39081)
0927 18:50:40.748611 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.748620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.749133 (+ 513us) server_negotiation.cc:407] Connection header received
0927 18:50:40.749382 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.749387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.749476 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.749585 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.750305 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42696: BlockingRecv error: recv got EOF from 127.0.0.1:42696 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 18:50:40.760227 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.757839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.31.26.253:39081)
0927 18:50:40.758095 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.758105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.758731 (+ 626us) server_negotiation.cc:407] Connection header received
0927 18:50:40.758966 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.758972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.759051 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.759169 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.759973 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42700: BlockingRecv error: recv got EOF from 127.0.0.1:42700 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 18:50:40.775167 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.772743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.31.26.253:39081)
0927 18:50:40.773010 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.773021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.773649 (+ 628us) server_negotiation.cc:407] Connection header received
0927 18:50:40.773894 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.773898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.773975 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.774093 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.774910 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42716: BlockingRecv error: recv got EOF from 127.0.0.1:42716 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:50:40.796787 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.794605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.31.26.253:39081)
0927 18:50:40.794867 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.794877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.795400 (+ 523us) server_negotiation.cc:407] Connection header received
0927 18:50:40.795614 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.795618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.795691 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.795798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.796554 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42722: BlockingRecv error: recv got EOF from 127.0.0.1:42722 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 18:50:40.834720 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.832208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.31.26.253:39081)
0927 18:50:40.832431 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.832437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.833044 (+ 607us) server_negotiation.cc:407] Connection header received
0927 18:50:40.833316 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.833320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.833397 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.833506 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.834445 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42724: BlockingRecv error: recv got EOF from 127.0.0.1:42724 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 18:50:40.904860 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:40.902405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.31.26.253:39081)
0927 18:50:40.902686 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:40.902696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:40.903322 (+ 626us) server_negotiation.cc:407] Connection header received
0927 18:50:40.903594 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:40.903598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:40.903684 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:40.903796 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:40.904636 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42734: BlockingRecv error: recv got EOF from 127.0.0.1:42734 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:50:40.966953 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41be32fd29284753b858a66ec1b417cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41be32fd29284753b858a66ec1b417cf" member_type: VOTER last_known_addr { host: "127.31.26.254" port: 44625 } } peers { permanent_uuid: "4fbc0bef76ee4c53b241f35db78c3088" member_type: VOTER last_known_addr { host: "127.31.26.253" port: 39081 } } peers { permanent_uuid: "b07bef3922714c4ab8aa659c7475d634" member_type: VOTER last_known_addr { host: "127.31.26.252" port: 46457 } } }
I20260927 18:50:40.967375 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41be32fd29284753b858a66ec1b417cf [sys.catalog]: This master's current role is: LEADER
W20260927 18:50:41.038686 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:41.036517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.31.26.253:39081)
0927 18:50:41.036766 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.036772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.037021 (+ 249us) server_negotiation.cc:407] Connection header received
0927 18:50:41.037315 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.037319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.037394 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.037485 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:41.038424 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42748: BlockingRecv error: recv got EOF from 127.0.0.1:42748 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260927 18:50:41.301314 22337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:41.299309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.31.26.253:39081)
0927 18:50:41.299489 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.299494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.299875 (+ 381us) server_negotiation.cc:407] Connection header received
0927 18:50:41.300043 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.300048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.300129 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.300255 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:41.301018 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42756: BlockingRecv error: recv got EOF from 127.0.0.1:42756 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:50:41.343600 22425 catalog_manager.cc:1259] Loaded cluster ID: 64e8d4c813414210843fcf8a6c103d27
I20260927 18:50:41.343782 22425 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: loading cluster ID for follower catalog manager: success
I20260927 18:50:41.346894 22425 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: acquiring CA information for follower catalog manager: success
I20260927 18:50:41.349176 22425 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b07bef3922714c4ab8aa659c7475d634: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:50:41.686270 22480 catalog_manager.cc:1259] Loaded cluster ID: 64e8d4c813414210843fcf8a6c103d27
I20260927 18:50:41.686430 22480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: loading cluster ID for follower catalog manager: success
I20260927 18:50:41.689352 22480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: acquiring CA information for follower catalog manager: success
I20260927 18:50:41.691493 22480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fbc0bef76ee4c53b241f35db78c3088: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:50:41.825348 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.817602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.31.26.253:39081)
0927 18:50:41.818280 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.818288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.818565 (+ 277us) server_negotiation.cc:407] Connection header received
0927 18:50:41.818822 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.818827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.818989 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.819095 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:41.819611 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.820569 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.822806 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.823266 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.823382 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:41.823990 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:41.824607 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:41.824741 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:41.824884 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 18:50:41.824959 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":341,"threads_started":1}
I20260927 18:50:41.838397 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.831937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.31.26.252:46457)
0927 18:50:41.832712 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.832718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.832741 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:50:41.832894 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.832901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.833073 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.833320 (+ 247us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:41.833821 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.834567 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.835939 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.836373 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.836513 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:41.837134 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:41.837734 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:41.837905 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:41.837966 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 18:50:41.838055 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":521,"threads_started":1}
I20260927 18:50:41.851724 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.844574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.31.26.253:39081)
0927 18:50:41.844813 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.844822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.846082 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 18:50:41.846316 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.846321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.846397 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.846523 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:41.847014 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.848036 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.849816 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.850252 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.850396 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:41.850845 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:41.851010 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:41.851112 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:41.851359 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 18:50:41.851444 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:50:41.858731 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.852022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.31.26.252:46457)
0927 18:50:41.852238 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.852244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.852723 (+ 479us) server_negotiation.cc:407] Connection header received
0927 18:50:41.853019 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.853027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.853154 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.853281 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:41.853861 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.854941 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.856760 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.857311 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.857443 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:41.857921 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:41.858052 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:41.858151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:41.858318 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 18:50:41.858423 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:50:41.858857 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.844523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.31.26.254:44625)
0927 18:50:41.845202 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:41.845209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:41.851242 (+ 6033us) server_negotiation.cc:407] Connection header received
0927 18:50:41.851465 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:41.851471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:41.851698 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:41.851890 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:41.853069 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.854176 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.856269 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:41.856955 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:41.857129 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:41.857670 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:41.858118 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:41.858228 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:41.858453 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:50:41.858542 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":431,"threads_started":1}
I20260927 18:50:41.864091 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 22190
I20260927 18:50:41.865507 22252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:41.977593 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 22190
W20260927 18:50:41.999105 22281 connection.cc:535] server connection from 127.0.0.1:58502 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:50:41.999415 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 22264
I20260927 18:50:42.000598 22326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:42.124063 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 22264
I20260927 18:50:42.147096 31851 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu pid 22339
I20260927 18:50:42.148216 22401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:50:42.258517 31851 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk9LG2U/build/asan/bin/kudu with pid 22339
2026-09-27T18:50:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4949 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9889 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (373326 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST